Jul 07 17:52:04.868053 ip-10-0-137-242 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 17:52:04.868065 ip-10-0-137-242 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 17:52:04.868074 ip-10-0-137-242 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 17:52:04.868408 ip-10-0-137-242 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 17:52:15.032679 ip-10-0-137-242 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 17:52:15.032695 ip-10-0-137-242 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b79c5e86a3b54291b4760b3753253543 -- Jul 07 17:54:14.645613 ip-10-0-137-242 systemd[1]: Starting Kubernetes Kubelet... Jul 07 17:54:14.995418 ip-10-0-137-242 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:54:14.995418 ip-10-0-137-242 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 17:54:14.995418 ip-10-0-137-242 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:54:14.995418 ip-10-0-137-242 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 17:54:14.995418 ip-10-0-137-242 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:54:14.996356 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:14.996188 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 17:54:15.000648 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000626 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:15.000648 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000648 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000652 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000655 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000658 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000667 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000670 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000672 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000677 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000680 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000682 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000685 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000688 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000691 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000694 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000696 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000699 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000701 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000704 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000707 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:15.000838 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000710 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000712 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000714 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000717 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000719 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000724 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000728 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000731 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000734 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000738 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000741 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000744 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000747 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000750 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000752 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000756 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000759 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000761 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000764 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000766 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:15.001351 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000769 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000771 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000774 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000776 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000779 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000781 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000785 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000787 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000790 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000792 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000795 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000797 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000800 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000802 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000804 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000807 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000811 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000813 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000816 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000818 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:15.001836 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000821 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000823 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000826 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000829 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000831 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000833 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000837 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000841 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000844 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000846 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000850 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000852 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000855 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000857 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000860 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000862 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000864 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000867 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000870 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000872 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:15.002377 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000875 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000877 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000880 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000885 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000888 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000891 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000894 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000898 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000902 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000905 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000908 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000911 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000914 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000917 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000920 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000923 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000926 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000928 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.000931 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:15.002848 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001695 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001702 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001705 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001708 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001711 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001713 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001716 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001719 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001721 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001724 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001726 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001729 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001731 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001734 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001737 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001740 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001742 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001745 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001747 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001750 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:15.003309 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001752 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001755 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001757 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001761 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001764 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001767 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001769 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001771 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001774 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001777 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001779 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001782 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001786 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001790 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001794 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001797 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001800 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001802 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001806 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:15.003823 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001809 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001812 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001815 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001818 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001820 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001823 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001826 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001829 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001831 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001834 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001837 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001839 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001842 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001844 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001847 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001850 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001853 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001855 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001858 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001861 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:15.004297 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001863 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001865 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001868 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001871 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001873 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001876 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001879 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001881 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001883 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001886 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001888 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001891 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001893 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001896 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001898 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001901 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001906 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001909 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001912 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:15.004851 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001915 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001918 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001921 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001923 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001925 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001928 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001930 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001933 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001936 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001938 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001941 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001943 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001946 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001948 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001951 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001953 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001956 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001959 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001964 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001966 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:15.005317 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.001969 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002391 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002396 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002399 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002402 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002405 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002407 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002410 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002413 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002415 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002418 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002420 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002423 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002425 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002427 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002430 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002432 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002435 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002437 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:15.005815 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002440 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002442 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002445 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002448 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002450 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002453 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002455 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002458 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002460 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002463 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002466 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002470 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002473 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002475 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002478 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002480 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002483 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002485 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002488 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002490 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:15.006270 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002492 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002495 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002497 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002500 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002502 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002505 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002507 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002511 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002513 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002516 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002518 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002521 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002523 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002527 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002529 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002532 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002534 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002536 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002539 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002541 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:15.006843 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002544 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002546 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002550 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002556 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002559 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002562 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002564 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002567 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002569 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002572 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002575 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002577 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002580 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002583 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002585 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002588 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002590 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002592 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002595 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002597 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:15.007411 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002600 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002602 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002604 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002607 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002609 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002612 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002616 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002620 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002622 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002625 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002628 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002632 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002635 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002637 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002640 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002644 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002647 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002649 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002652 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:15.007964 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002654 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.002657 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002746 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002757 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002767 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002774 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002780 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002784 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002789 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002793 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002796 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002800 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002803 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002807 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002811 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002814 2557 flags.go:64] FLAG: --cgroup-root="" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002817 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002820 2557 flags.go:64] FLAG: --client-ca-file="" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002823 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002826 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002829 2557 flags.go:64] FLAG: --cluster-domain="" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002833 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002836 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002839 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 17:54:15.008525 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002842 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002846 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002849 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002853 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002858 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002861 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002864 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002868 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002872 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002875 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002879 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002882 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002885 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002888 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002891 2557 flags.go:64] FLAG: --enable-server="true" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002893 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002901 2557 flags.go:64] FLAG: --event-burst="100" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002904 2557 flags.go:64] FLAG: --event-qps="50" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002907 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002910 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002913 2557 flags.go:64] FLAG: --eviction-hard="" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002917 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002919 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002922 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002925 2557 flags.go:64] FLAG: --eviction-soft="" Jul 07 17:54:15.009177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002928 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002931 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002934 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002937 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002940 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002943 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002946 2557 flags.go:64] FLAG: --feature-gates="" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002949 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002952 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002955 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002958 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002962 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002965 2557 flags.go:64] FLAG: --help="false" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002968 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002972 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002975 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002978 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002981 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002984 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002987 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002990 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002993 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002996 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 17:54:15.009890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.002999 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003002 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003005 2557 flags.go:64] FLAG: --kube-reserved="" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003008 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003010 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003013 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003016 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003019 2557 flags.go:64] FLAG: --lock-file="" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003022 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003025 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003028 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003033 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003036 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003039 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003042 2557 flags.go:64] FLAG: --logging-format="text" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003044 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003048 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003050 2557 flags.go:64] FLAG: --manifest-url="" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003053 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003057 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003061 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003065 2557 flags.go:64] FLAG: --max-pods="110" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003068 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003071 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003074 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 17:54:15.010591 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003078 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003081 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003084 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003087 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003094 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003098 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003101 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003104 2557 flags.go:64] FLAG: --pod-cidr="" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003107 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003112 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003115 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003118 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003120 2557 flags.go:64] FLAG: --port="10250" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003123 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003126 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e526de82a6878a8e" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003129 2557 flags.go:64] FLAG: --qos-reserved="" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003132 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003135 2557 flags.go:64] FLAG: --register-node="true" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003138 2557 flags.go:64] FLAG: --register-with-taints="" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003142 2557 flags.go:64] FLAG: --registry-burst="10" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003144 2557 flags.go:64] FLAG: --registry-qps="5" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003147 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003150 2557 flags.go:64] FLAG: --reserved-memory="" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003153 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003157 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 17:54:15.011553 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003160 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003163 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003165 2557 flags.go:64] FLAG: --runonce="false" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003170 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003173 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003176 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003179 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003183 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003186 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003189 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003192 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003194 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003197 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003200 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003203 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003206 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003208 2557 flags.go:64] FLAG: --system-cgroups="" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003212 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003217 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003219 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003222 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003225 2557 flags.go:64] FLAG: --tls-min-version="" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003229 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003232 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003234 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 17:54:15.012236 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003237 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003240 2557 flags.go:64] FLAG: --v="2" Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003244 2557 flags.go:64] FLAG: --version="false" Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003248 2557 flags.go:64] FLAG: --vmodule="" Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003252 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.003255 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003396 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003400 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003403 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003406 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003410 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003414 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003416 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003421 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003424 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003427 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003430 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003433 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003435 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003438 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003440 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:15.012886 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003443 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003445 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003448 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003450 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003452 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003455 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003457 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003460 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003462 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003464 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003467 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003469 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003473 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003477 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003479 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003482 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003485 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003488 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003492 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:15.013416 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003495 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003497 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003501 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003504 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003506 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003509 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003512 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003515 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003518 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003520 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003523 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003525 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003528 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003530 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003532 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003535 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003537 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003540 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003542 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003544 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:15.013875 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003547 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003549 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003551 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003554 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003556 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003559 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003561 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003564 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003566 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003569 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003571 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003574 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003576 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003578 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003582 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003585 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003588 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003592 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003594 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003597 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:15.014368 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003599 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003602 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003605 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003608 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003610 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003613 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003615 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003617 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003620 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003623 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003625 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003627 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003630 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003632 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003635 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003637 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003639 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003642 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003645 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:15.014890 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003647 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003651 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003654 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003656 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003659 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.003662 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.004219 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.012619 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.012635 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.013439 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.013448 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013508 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013514 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013518 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:15.015360 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013521 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013524 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013526 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013529 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013532 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013535 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013539 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013542 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013546 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013548 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013551 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013553 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013556 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013559 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013562 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013566 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013569 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013572 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013575 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:15.015712 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013578 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013581 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013583 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013586 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013588 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013591 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013593 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013596 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013599 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013602 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013605 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013607 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013610 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013612 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013615 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013617 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013620 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013622 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013625 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:15.016169 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013627 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013630 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013632 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013635 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013638 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013641 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013644 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013646 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013649 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013652 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013654 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013657 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013659 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013662 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013665 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013668 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013670 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013673 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013675 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013678 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:15.016652 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013681 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013683 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013686 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013688 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013691 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013693 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013696 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013700 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013703 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013705 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013708 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013711 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013714 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013716 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013719 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013721 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013724 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013726 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013730 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:15.017153 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013733 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013735 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013737 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013740 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013742 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013745 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013748 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013750 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013753 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013756 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013758 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013761 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013763 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013766 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013768 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013770 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013773 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013775 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:15.017641 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013778 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.013783 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013898 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013903 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013906 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013909 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013911 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013914 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013917 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013920 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013923 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013925 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013928 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013931 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013935 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:15.018076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013938 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013940 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013943 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013946 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013948 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013951 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013953 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013956 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013958 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013961 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013963 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013967 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013971 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013975 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013978 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013981 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013984 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013986 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013989 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:15.018500 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013991 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013994 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013996 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.013999 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014001 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014004 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014007 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014009 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014011 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014014 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014017 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014020 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014024 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014027 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014030 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014033 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014036 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014038 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014041 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014043 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:15.018955 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014046 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014049 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014051 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014054 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014056 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014059 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014063 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014067 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014069 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014072 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014074 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014076 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014079 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014081 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014084 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014086 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014089 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014091 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014094 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014096 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:15.019503 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014099 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014102 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014105 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014107 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014110 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014112 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014116 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014118 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014120 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014123 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014125 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014128 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014130 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014133 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014135 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014138 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014140 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014143 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014147 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:15.019989 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014149 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014152 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014154 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014157 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014159 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014162 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014165 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:15.014167 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.014172 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.014314 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.017267 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.018498 2557 server.go:1019] "Starting client certificate rotation" Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.018596 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:54:15.020465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.018638 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:54:15.020965 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.020949 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:54:15.022339 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.022315 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:54:15.030393 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.030374 2557 log.go:25] "Validated CRI v1 runtime API" Jul 07 17:54:15.034554 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.034535 2557 log.go:25] "Validated CRI v1 image API" Jul 07 17:54:15.035398 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.035380 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 17:54:15.037865 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.037845 2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b77d71af-98a2-483b-87ad-c024b2f9cb63:/dev/nvme0n1p4 c2498f8f-b99a-4ba5-b892-de0f7c24c576:/dev/nvme0n1p3] Jul 07 17:54:15.037942 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.037865 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 17:54:15.043708 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.043595 2557 manager.go:216] Machine: {Timestamp:2026-07-07 17:54:15.04308025 +0000 UTC m=+0.296196841 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095395 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d80bf6b779089ae3b9ec6e8aaf406 SystemUUID:ec2d80bf-6b77-9089-ae3b-9ec6e8aaf406 BootID:b79c5e86-a3b5-4291-b476-0b3753253543 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b5:4c:00:01:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b5:4c:00:01:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:cc:65:69:b1:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 17:54:15.043708 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.043696 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 17:54:15.043838 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.043776 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 17:54:15.045262 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.045228 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 17:54:15.045419 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.045266 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 17:54:15.046585 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.046574 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 17:54:15.046625 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.046588 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 17:54:15.046625 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.046596 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:54:15.047107 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.047097 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:54:15.047504 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.047489 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:54:15.048216 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.048204 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 17:54:15.050940 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.050929 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:54:15.051066 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.051057 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 17:54:15.051107 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.051086 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 07 17:54:15.051107 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.051096 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 17:54:15.051160 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.051111 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 17:54:15.051160 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.051120 2557 kubelet.go:398] "Adding apiserver pod source" Jul 07 17:54:15.051160 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.051129 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 17:54:15.051637 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.051625 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:54:15.051686 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.051647 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:54:15.053775 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.053761 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 17:54:15.055429 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.055414 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 17:54:15.055470 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.055448 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 17:54:15.056402 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056390 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056406 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056412 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056417 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056426 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056434 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056447 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056455 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056462 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056467 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 17:54:15.056471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056476 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 17:54:15.056734 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056490 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 17:54:15.057002 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056987 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 17:54:15.057002 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.056999 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 17:54:15.060387 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.060375 2557 server.go:1268] "Started kubelet" Jul 07 17:54:15.060494 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.060458 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 17:54:15.060568 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.060472 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 17:54:15.060693 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.060677 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 17:54:15.061306 ip-10-0-137-242 systemd[1]: Started Kubernetes Kubelet. Jul 07 17:54:15.062351 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.062289 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 17:54:15.062351 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.062342 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 17:54:15.062461 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.062388 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 17:54:15.062542 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.062527 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-242.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 17:54:15.062974 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.062961 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 07 17:54:15.064887 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.064866 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 17:54:15.071335 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.070522 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18c013772fa8e7c6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-07-07 17:54:15.060350918 +0000 UTC m=+0.313467493,LastTimestamp:2026-07-07 17:54:15.060350918 +0000 UTC m=+0.313467493,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Jul 07 17:54:15.072414 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.072396 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 17:54:15.072506 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.072418 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 17:54:15.072572 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.072514 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.073402 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.073382 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 17:54:15.073572 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.073553 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 17:54:15.073572 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.073573 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 17:54:15.073674 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.073645 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 07 17:54:15.073674 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.073652 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 07 17:54:15.073750 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.073704 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 17:54:15.073889 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.073868 2557 factory.go:55] Registering systemd factory Jul 07 17:54:15.073971 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.073893 2557 factory.go:223] Registration of the systemd container factory successfully Jul 07 17:54:15.074123 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.074108 2557 factory.go:153] Registering CRI-O factory Jul 07 17:54:15.074123 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.074122 2557 factory.go:223] Registration of the crio container factory successfully Jul 07 17:54:15.074231 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.074139 2557 factory.go:103] Registering Raw factory Jul 07 17:54:15.074231 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.074148 2557 manager.go:1196] Started watching for new ooms in manager Jul 07 17:54:15.074673 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.074655 2557 manager.go:349] Starting recovery of all containers Jul 07 17:54:15.074871 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.074847 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 17:54:15.084906 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.084871 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 17:54:15.086973 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.086851 2557 manager.go:354] Recovery completed Jul 07 17:54:15.088096 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.088072 2557 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 07 17:54:15.091105 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.091089 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:15.093053 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.093033 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:15.093120 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.093068 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:15.093120 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.093080 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:15.093696 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.093675 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 17:54:15.093696 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.093689 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:15.093696 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.093695 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 17:54:15.093857 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.093714 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:54:15.094376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.094357 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:15.094432 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.094388 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:15.094432 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.094401 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:15.095058 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.095044 2557 policy_none.go:49] "None policy: Start" Jul 07 17:54:15.095100 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.095064 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 17:54:15.095100 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.095076 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 17:54:15.096179 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.096168 2557 policy_none.go:47] "Start" Jul 07 17:54:15.102685 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.102617 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18c01377319be570 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-242.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-07-07 17:54:15.093052784 +0000 UTC m=+0.346169361,LastTimestamp:2026-07-07 17:54:15.093052784 +0000 UTC m=+0.346169361,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Jul 07 17:54:15.106158 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.106099 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18c01377319c37c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-242.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-07-07 17:54:15.093073856 +0000 UTC m=+0.346190428,LastTimestamp:2026-07-07 17:54:15.093073856 +0000 UTC m=+0.346190428,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Jul 07 17:54:15.114570 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.114508 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18c01377319c5e24 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-242.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-07-07 17:54:15.093083684 +0000 UTC m=+0.346200255,LastTimestamp:2026-07-07 17:54:15.093083684 +0000 UTC m=+0.346200255,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Jul 07 17:54:15.118632 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.118548 2557 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-242.ec2.internal.18c01377319be570\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18c01377319be570 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-242.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-07-07 17:54:15.093052784 +0000 UTC m=+0.346169361,LastTimestamp:2026-07-07 17:54:15.094373614 +0000 UTC m=+0.347490191,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Jul 07 17:54:15.126315 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.126121 2557 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-242.ec2.internal.18c01377319c37c0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18c01377319c37c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-242.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-07-07 17:54:15.093073856 +0000 UTC m=+0.346190428,LastTimestamp:2026-07-07 17:54:15.094395759 +0000 UTC m=+0.347512330,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Jul 07 17:54:15.129180 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.129113 2557 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-242.ec2.internal.18c01377319c5e24\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18c01377319c5e24 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-242.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-07-07 17:54:15.093083684 +0000 UTC m=+0.346200255,LastTimestamp:2026-07-07 17:54:15.094405208 +0000 UTC m=+0.347521778,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Jul 07 17:54:15.136900 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.136884 2557 manager.go:337] "Starting Device Plugin manager" Jul 07 17:54:15.137001 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.136933 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 17:54:15.137001 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.136947 2557 server.go:87] "Starting device plugin registration server" Jul 07 17:54:15.137254 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.137241 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 17:54:15.137313 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.137256 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 17:54:15.137375 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.137344 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 17:54:15.137446 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.137431 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 17:54:15.137490 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.137447 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 17:54:15.138188 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.138167 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 17:54:15.138247 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.138227 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.141298 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.141215 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18c01377345a1cc8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-07-07 17:54:15.139073224 +0000 UTC m=+0.392189783,LastTimestamp:2026-07-07 17:54:15.139073224 +0000 UTC m=+0.392189783,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Jul 07 17:54:15.159563 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.159542 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnpv4" Jul 07 17:54:15.169380 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.169352 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnpv4" Jul 07 17:54:15.215959 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.215916 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 17:54:15.217389 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.217370 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 17:54:15.217509 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.217414 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 17:54:15.217509 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.217457 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 17:54:15.217605 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.217500 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 17:54:15.228410 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.228382 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:15.238373 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.238358 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:15.239232 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.239216 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:15.239343 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.239248 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:15.239343 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.239277 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:15.239343 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.239307 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.249252 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.249202 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.249252 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.249225 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-242.ec2.internal\": node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.265569 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.265542 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.318529 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.318494 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal"] Jul 07 17:54:15.318692 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.318573 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:15.320598 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.320579 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:15.320685 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.320612 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:15.320685 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.320626 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:15.321853 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.321840 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:15.322043 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.322029 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.322093 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.322074 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:15.322661 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.322645 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:15.322748 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.322672 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:15.322748 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.322678 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:15.322748 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.322710 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:15.322748 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.322724 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:15.322933 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.322684 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:15.323987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.323970 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.324070 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.323998 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:15.325047 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.325032 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:15.325126 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.325060 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:15.325126 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.325074 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:15.350017 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.349990 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-242.ec2.internal\" not found" node="ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.354697 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.354682 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-242.ec2.internal\" not found" node="ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.366244 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.366225 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.375625 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.375602 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca7e154a214d139ef80f0ff105532d9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ca7e154a214d139ef80f0ff105532d9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.375707 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.375634 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/79639d5bc5ecaca9c381236434449ff7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"79639d5bc5ecaca9c381236434449ff7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.375707 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.375692 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca7e154a214d139ef80f0ff105532d9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ca7e154a214d139ef80f0ff105532d9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.467353 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.467304 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.476424 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.476403 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca7e154a214d139ef80f0ff105532d9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ca7e154a214d139ef80f0ff105532d9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.476501 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.476441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca7e154a214d139ef80f0ff105532d9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ca7e154a214d139ef80f0ff105532d9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.476501 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.476469 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/79639d5bc5ecaca9c381236434449ff7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"79639d5bc5ecaca9c381236434449ff7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.476720 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.476508 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca7e154a214d139ef80f0ff105532d9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ca7e154a214d139ef80f0ff105532d9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.476812 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.476775 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca7e154a214d139ef80f0ff105532d9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ca7e154a214d139ef80f0ff105532d9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.476999 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.476977 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/79639d5bc5ecaca9c381236434449ff7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"79639d5bc5ecaca9c381236434449ff7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.567963 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.567861 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.653458 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.653414 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.658096 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.657746 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.668796 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.668772 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.769416 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.769379 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.871303 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:15.871229 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Jul 07 17:54:15.961700 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.961672 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:15.972764 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.972742 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.985376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.985356 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:54:15.986946 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.986932 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Jul 07 17:54:15.997462 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:15.997440 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:54:16.007751 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.007729 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:16.018765 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.018743 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 17:54:16.018890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.018868 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 17:54:16.018933 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.018904 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 17:54:16.018978 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.018924 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 17:54:16.053641 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.053604 2557 apiserver.go:52] "Watching apiserver" Jul 07 17:54:16.072823 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.072788 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 17:54:16.078302 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.078276 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 17:54:16.078728 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.078698 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-9glvk","kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d","openshift-cluster-node-tuning-operator/tuned-nsg2b","openshift-image-registry/node-ca-rnnrw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal","openshift-network-diagnostics/network-check-target-dq27h","openshift-network-operator/iptables-alerter-jn24t","kube-system/global-pull-secret-syncer-6s26g","openshift-dns/node-resolver-52tkx","openshift-multus/multus-additional-cni-plugins-h8npv","openshift-multus/multus-sksph","openshift-multus/network-metrics-daemon-7zfpw","openshift-ovn-kubernetes/ovnkube-node-j7v76"] Jul 07 17:54:16.080435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.080410 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:16.080532 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.080482 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:16.081576 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.081560 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.082910 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.082889 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.083935 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.083888 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 17:54:16.083935 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.083928 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 17:54:16.084199 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.084183 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.085520 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.085501 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:16.085827 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.085810 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8zjgn\"" Jul 07 17:54:16.085964 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.085947 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 17:54:16.086046 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.085948 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 17:54:16.086046 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.085953 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 17:54:16.086364 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.086347 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 17:54:16.086461 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.086363 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:54:16.086777 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.086764 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.087942 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.087925 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.090930 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.090274 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sksph" Jul 07 17:54:16.090930 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.090739 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:16.090930 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.090899 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 17:54:16.091207 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.090951 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kqgs8\"" Jul 07 17:54:16.091207 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.091113 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 17:54:16.091207 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.091180 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 17:54:16.091397 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.091366 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 17:54:16.091397 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.091373 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vlwzk\"" Jul 07 17:54:16.091559 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.091491 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dg7d8\"" Jul 07 17:54:16.091764 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.091735 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 17:54:16.092009 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.091934 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 17:54:16.092009 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.091980 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 17:54:16.092338 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.092289 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 17:54:16.092438 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.092425 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 17:54:16.092519 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.092466 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lww89\"" Jul 07 17:54:16.092519 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.092473 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 17:54:16.092519 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.092470 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 17:54:16.092674 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.092649 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4spjk\"" Jul 07 17:54:16.093172 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.093152 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 17:54:16.093347 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.093305 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 17:54:16.093470 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.093450 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:16.093569 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.093552 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tkwrm\"" Jul 07 17:54:16.093662 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.093642 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 17:54:16.093761 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.093721 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.094992 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.094972 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.096559 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.096260 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:16.096559 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.096378 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:16.097520 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.097502 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.097598 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.097558 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:16.098705 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.098687 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 17:54:16.098776 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.098739 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 17:54:16.098988 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.098967 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tgc8\"" Jul 07 17:54:16.098988 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.098979 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mptkz\"" Jul 07 17:54:16.099118 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.099021 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 17:54:16.099118 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.098979 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 17:54:16.099118 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.099089 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 17:54:16.124265 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.124086 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tch72" Jul 07 17:54:16.133461 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.133437 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tch72" Jul 07 17:54:16.171437 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.171365 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 17:49:15 +0000 UTC" deadline="2028-01-29 00:01:51.146812312 +0000 UTC" Jul 07 17:54:16.171437 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.171434 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13686h7m34.975381777s" Jul 07 17:54:16.175456 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.175415 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 17:54:16.180794 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.180766 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-host\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.180903 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.180818 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-tuned\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.180903 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.180864 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-kubelet-config\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.180903 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.180895 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-os-release\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.181075 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.180918 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9jc\" (UniqueName: \"kubernetes.io/projected/bfcd6894-b47b-4ecd-9dc2-38235226086c-kube-api-access-cb9jc\") pod \"node-ca-rnnrw\" (UID: \"bfcd6894-b47b-4ecd-9dc2-38235226086c\") " pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.181075 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.180947 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-run-netns\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181075 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.180991 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.181075 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181034 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-sysconfig\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.181075 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181063 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpf96\" (UniqueName: \"kubernetes.io/projected/2617985b-2645-4ad2-b29c-102d5b3acbd4-kube-api-access-dpf96\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181090 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-var-lib-cni-multus\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181116 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f277f07-7a2c-4881-b1ba-aefff89d1279-ovnkube-config\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181153 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-var-lib-cni-bin\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181194 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-sys-fs\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181220 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-slash\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181246 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-run-openvswitch\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181273 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d258f463-231f-41f9-89f2-e2077274343f-iptables-alerter-script\") pod \"iptables-alerter-jn24t\" (UID: \"d258f463-231f-41f9-89f2-e2077274343f\") " pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181296 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d258f463-231f-41f9-89f2-e2077274343f-host-slash\") pod \"iptables-alerter-jn24t\" (UID: \"d258f463-231f-41f9-89f2-e2077274343f\") " pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181330 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-run-systemd\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181429 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-run-ovn-kubernetes\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181456 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f277f07-7a2c-4881-b1ba-aefff89d1279-env-overrides\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181496 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f277f07-7a2c-4881-b1ba-aefff89d1279-ovnkube-script-lib\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181539 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qz7w\" (UniqueName: \"kubernetes.io/projected/d258f463-231f-41f9-89f2-e2077274343f-kube-api-access-2qz7w\") pod \"iptables-alerter-jn24t\" (UID: \"d258f463-231f-41f9-89f2-e2077274343f\") " pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181579 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-systemd\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181602 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-cnibin\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181618 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-hostroot\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181633 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-modprobe-d\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181652 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-socket-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181695 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-systemd-units\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-var-lib-openvswitch\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181752 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-cni-bin\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181792 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-run\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181827 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f2fe74e-03fd-4704-a472-8965dd5d4230-konnectivity-ca\") pod \"konnectivity-agent-9glvk\" (UID: \"9f2fe74e-03fd-4704-a472-8965dd5d4230\") " pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181854 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-run-multus-certs\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181880 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-system-cni-dir\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181913 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05818555-acf1-4dff-9440-e639a27413b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.181944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181937 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-sysctl-conf\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.181963 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-os-release\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182013 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-var-lib-kubelet\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182045 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bfcd6894-b47b-4ecd-9dc2-38235226086c-serviceca\") pod \"node-ca-rnnrw\" (UID: \"bfcd6894-b47b-4ecd-9dc2-38235226086c\") " pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182084 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182115 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-log-socket\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182138 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-sys\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182162 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-dbus\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182188 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05818555-acf1-4dff-9440-e639a27413b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182215 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-kubernetes\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182239 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-run-k8s-cni-cncf-io\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182264 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-run-netns\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182304 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182350 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffwtc\" (UniqueName: \"kubernetes.io/projected/a713dd27-8e40-4565-8c88-4378a67efc21-kube-api-access-ffwtc\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182368 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1fffc956-ac04-4a88-a353-a9aa99e3a8a3-hosts-file\") pod \"node-resolver-52tkx\" (UID: \"1fffc956-ac04-4a88-a353-a9aa99e3a8a3\") " pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182383 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-device-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.182653 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182399 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-etc-openvswitch\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182426 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brw2\" (UniqueName: \"kubernetes.io/projected/d207b749-497b-4625-9510-3c0855f6b245-kube-api-access-8brw2\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182443 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2617985b-2645-4ad2-b29c-102d5b3acbd4-tmp\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182462 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdsnm\" (UniqueName: \"kubernetes.io/projected/8d593cf7-aaaa-48f9-8f72-c823d8378efe-kube-api-access-mdsnm\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182477 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-system-cni-dir\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182490 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-cnibin\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182505 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-socket-dir-parent\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182530 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-daemon-config\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182589 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fffc956-ac04-4a88-a353-a9aa99e3a8a3-tmp-dir\") pod \"node-resolver-52tkx\" (UID: \"1fffc956-ac04-4a88-a353-a9aa99e3a8a3\") " pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182627 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182660 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkh9f\" (UniqueName: \"kubernetes.io/projected/4f277f07-7a2c-4881-b1ba-aefff89d1279-kube-api-access-wkh9f\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlfg\" (UniqueName: \"kubernetes.io/projected/05818555-acf1-4dff-9440-e639a27413b9-kube-api-access-qwlfg\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182734 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-sysctl-d\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182771 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f2fe74e-03fd-4704-a472-8965dd5d4230-agent-certs\") pod \"konnectivity-agent-9glvk\" (UID: \"9f2fe74e-03fd-4704-a472-8965dd5d4230\") " pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182799 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfcd6894-b47b-4ecd-9dc2-38235226086c-host\") pod \"node-ca-rnnrw\" (UID: \"bfcd6894-b47b-4ecd-9dc2-38235226086c\") " pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182826 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-registration-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.183376 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182857 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-run-ovn\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182902 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182936 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182961 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-lib-modules\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.182986 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-kubelet\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183009 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-node-log\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183036 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f277f07-7a2c-4881-b1ba-aefff89d1279-ovn-node-metrics-cert\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183060 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-cni-dir\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183084 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d593cf7-aaaa-48f9-8f72-c823d8378efe-cni-binary-copy\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183147 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-conf-dir\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183177 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05818555-acf1-4dff-9440-e639a27413b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183203 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-var-lib-kubelet\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183229 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-etc-kubernetes\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183256 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdmzj\" (UniqueName: \"kubernetes.io/projected/1fffc956-ac04-4a88-a353-a9aa99e3a8a3-kube-api-access-cdmzj\") pod \"node-resolver-52tkx\" (UID: \"1fffc956-ac04-4a88-a353-a9aa99e3a8a3\") " pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183285 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-cni-netd\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.183908 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.183311 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:16.230098 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.230064 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79639d5bc5ecaca9c381236434449ff7.slice/crio-f481044dbb7bf8447ab0f185915fe3a895d082b6b4a6c4d13a0fdd8a216e133f WatchSource:0}: Error finding container f481044dbb7bf8447ab0f185915fe3a895d082b6b4a6c4d13a0fdd8a216e133f: Status 404 returned error can't find the container with id f481044dbb7bf8447ab0f185915fe3a895d082b6b4a6c4d13a0fdd8a216e133f Jul 07 17:54:16.230355 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.230316 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7e154a214d139ef80f0ff105532d9a.slice/crio-42004c8c60437a6f64e0a63e66f4e13be3116705c4b9ff11dde5694daee66190 WatchSource:0}: Error finding container 42004c8c60437a6f64e0a63e66f4e13be3116705c4b9ff11dde5694daee66190: Status 404 returned error can't find the container with id 42004c8c60437a6f64e0a63e66f4e13be3116705c4b9ff11dde5694daee66190 Jul 07 17:54:16.234962 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.234948 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:54:16.284267 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.284267 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284270 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-log-socket\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284289 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-sys\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-dbus\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284321 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05818555-acf1-4dff-9440-e639a27413b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284350 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-kubernetes\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284373 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-log-socket\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284376 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-run-k8s-cni-cncf-io\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284414 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-run-k8s-cni-cncf-io\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284422 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-sys\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284435 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-run-netns\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284457 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284464 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284497 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-kubernetes\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284490 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-dbus\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284515 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-run-netns\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284524 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffwtc\" (UniqueName: \"kubernetes.io/projected/a713dd27-8e40-4565-8c88-4378a67efc21-kube-api-access-ffwtc\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284561 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1fffc956-ac04-4a88-a353-a9aa99e3a8a3-hosts-file\") pod \"node-resolver-52tkx\" (UID: \"1fffc956-ac04-4a88-a353-a9aa99e3a8a3\") " pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.284564 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284569 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284581 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-device-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-etc-openvswitch\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284639 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brw2\" (UniqueName: \"kubernetes.io/projected/d207b749-497b-4625-9510-3c0855f6b245-kube-api-access-8brw2\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284646 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1fffc956-ac04-4a88-a353-a9aa99e3a8a3-hosts-file\") pod \"node-resolver-52tkx\" (UID: \"1fffc956-ac04-4a88-a353-a9aa99e3a8a3\") " pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284666 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-etc-openvswitch\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284667 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2617985b-2645-4ad2-b29c-102d5b3acbd4-tmp\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284641 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-device-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284709 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdsnm\" (UniqueName: \"kubernetes.io/projected/8d593cf7-aaaa-48f9-8f72-c823d8378efe-kube-api-access-mdsnm\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-system-cni-dir\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284745 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-cnibin\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284760 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-socket-dir-parent\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284780 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-daemon-config\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284802 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fffc956-ac04-4a88-a353-a9aa99e3a8a3-tmp-dir\") pod \"node-resolver-52tkx\" (UID: \"1fffc956-ac04-4a88-a353-a9aa99e3a8a3\") " pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284818 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284825 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-system-cni-dir\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284834 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkh9f\" (UniqueName: \"kubernetes.io/projected/4f277f07-7a2c-4881-b1ba-aefff89d1279-kube-api-access-wkh9f\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.285465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlfg\" (UniqueName: \"kubernetes.io/projected/05818555-acf1-4dff-9440-e639a27413b9-kube-api-access-qwlfg\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284938 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-socket-dir-parent\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284987 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-sysctl-d\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.284991 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285007 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-cnibin\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285044 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f2fe74e-03fd-4704-a472-8965dd5d4230-agent-certs\") pod \"konnectivity-agent-9glvk\" (UID: \"9f2fe74e-03fd-4704-a472-8965dd5d4230\") " pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285066 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfcd6894-b47b-4ecd-9dc2-38235226086c-host\") pod \"node-ca-rnnrw\" (UID: \"bfcd6894-b47b-4ecd-9dc2-38235226086c\") " pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285089 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-registration-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285113 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-run-ovn\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285148 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285167 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fffc956-ac04-4a88-a353-a9aa99e3a8a3-tmp-dir\") pod \"node-resolver-52tkx\" (UID: \"1fffc956-ac04-4a88-a353-a9aa99e3a8a3\") " pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285175 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05818555-acf1-4dff-9440-e639a27413b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285192 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285204 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-sysctl-d\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285230 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-run-ovn\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285242 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-registration-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285246 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285253 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfcd6894-b47b-4ecd-9dc2-38235226086c-host\") pod \"node-ca-rnnrw\" (UID: \"bfcd6894-b47b-4ecd-9dc2-38235226086c\") " pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.286161 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285282 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-lib-modules\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-kubelet\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285320 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-node-log\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285338 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f277f07-7a2c-4881-b1ba-aefff89d1279-ovn-node-metrics-cert\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285355 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-cni-dir\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285381 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d593cf7-aaaa-48f9-8f72-c823d8378efe-cni-binary-copy\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285405 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285417 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-node-log\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285427 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-conf-dir\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285462 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05818555-acf1-4dff-9440-e639a27413b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285486 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-var-lib-kubelet\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285519 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-etc-kubernetes\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285529 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-lib-modules\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285545 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdmzj\" (UniqueName: \"kubernetes.io/projected/1fffc956-ac04-4a88-a353-a9aa99e3a8a3-kube-api-access-cdmzj\") pod \"node-resolver-52tkx\" (UID: \"1fffc956-ac04-4a88-a353-a9aa99e3a8a3\") " pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285574 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-cni-netd\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285586 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-cni-dir\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285601 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285626 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-host\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.287235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285638 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-daemon-config\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285650 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-tuned\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285664 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-etc-kubernetes\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285676 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-kubelet-config\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285705 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-os-release\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.285749 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285769 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9jc\" (UniqueName: \"kubernetes.io/projected/bfcd6894-b47b-4ecd-9dc2-38235226086c-kube-api-access-cb9jc\") pod \"node-ca-rnnrw\" (UID: \"bfcd6894-b47b-4ecd-9dc2-38235226086c\") " pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285797 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-run-netns\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.285819 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs podName:d207b749-497b-4625-9510-3c0855f6b245 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:16.785797833 +0000 UTC m=+2.038914415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs") pod "network-metrics-daemon-7zfpw" (UID: "d207b749-497b-4625-9510-3c0855f6b245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285833 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-cni-netd\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285840 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-run-netns\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285882 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d593cf7-aaaa-48f9-8f72-c823d8378efe-cni-binary-copy\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285898 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-host\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285624 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-kubelet\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.285917 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285952 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-var-lib-kubelet\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.288146 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.285958 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret podName:1cd8e48b-8053-42a2-b2fb-777e20a75ed1 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:16.785941775 +0000 UTC m=+2.039058337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret") pod "global-pull-secret-syncer-6s26g" (UID: "1cd8e48b-8053-42a2-b2fb-777e20a75ed1") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.285999 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-multus-conf-dir\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286013 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-os-release\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286057 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-kubelet-config\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286084 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-sysconfig\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286108 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpf96\" (UniqueName: \"kubernetes.io/projected/2617985b-2645-4ad2-b29c-102d5b3acbd4-kube-api-access-dpf96\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286131 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-var-lib-cni-multus\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286154 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f277f07-7a2c-4881-b1ba-aefff89d1279-ovnkube-config\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286164 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05818555-acf1-4dff-9440-e639a27413b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286183 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-var-lib-cni-bin\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286226 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-sys-fs\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286246 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-var-lib-cni-bin\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286255 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-slash\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286282 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-run-openvswitch\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286299 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-sysconfig\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286309 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d258f463-231f-41f9-89f2-e2077274343f-iptables-alerter-script\") pod \"iptables-alerter-jn24t\" (UID: \"d258f463-231f-41f9-89f2-e2077274343f\") " pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286338 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-var-lib-cni-multus\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.288791 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286362 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d258f463-231f-41f9-89f2-e2077274343f-host-slash\") pod \"iptables-alerter-jn24t\" (UID: \"d258f463-231f-41f9-89f2-e2077274343f\") " pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286386 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-slash\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286391 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-run-systemd\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286404 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-sys-fs\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286442 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-run-ovn-kubernetes\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286464 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-run-openvswitch\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286473 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f277f07-7a2c-4881-b1ba-aefff89d1279-env-overrides\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286531 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-run-ovn-kubernetes\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286566 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-run-systemd\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286622 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f277f07-7a2c-4881-b1ba-aefff89d1279-ovnkube-script-lib\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286653 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d258f463-231f-41f9-89f2-e2077274343f-host-slash\") pod \"iptables-alerter-jn24t\" (UID: \"d258f463-231f-41f9-89f2-e2077274343f\") " pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286666 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qz7w\" (UniqueName: \"kubernetes.io/projected/d258f463-231f-41f9-89f2-e2077274343f-kube-api-access-2qz7w\") pod \"iptables-alerter-jn24t\" (UID: \"d258f463-231f-41f9-89f2-e2077274343f\") " pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286845 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-systemd\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286875 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-cnibin\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286901 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-hostroot\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286919 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-systemd\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286928 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-modprobe-d\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.289435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286935 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f277f07-7a2c-4881-b1ba-aefff89d1279-ovnkube-config\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.286997 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-socket-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287050 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-systemd-units\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287000 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-hostroot\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287099 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-modprobe-d\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287104 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-var-lib-openvswitch\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287134 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-cni-bin\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287139 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-cnibin\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287185 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-systemd-units\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287198 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-var-lib-openvswitch\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287207 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f277f07-7a2c-4881-b1ba-aefff89d1279-ovnkube-script-lib\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287231 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-run\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287234 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f277f07-7a2c-4881-b1ba-aefff89d1279-host-cni-bin\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287257 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f2fe74e-03fd-4704-a472-8965dd5d4230-konnectivity-ca\") pod \"konnectivity-agent-9glvk\" (UID: \"9f2fe74e-03fd-4704-a472-8965dd5d4230\") " pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287355 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a713dd27-8e40-4565-8c88-4378a67efc21-socket-dir\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287389 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-run-multus-certs\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287432 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f277f07-7a2c-4881-b1ba-aefff89d1279-env-overrides\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287493 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-system-cni-dir\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.289987 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287458 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-run\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287533 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05818555-acf1-4dff-9440-e639a27413b9-system-cni-dir\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287582 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05818555-acf1-4dff-9440-e639a27413b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287689 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-run-multus-certs\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-sysctl-conf\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-os-release\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287804 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-var-lib-kubelet\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287817 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f2fe74e-03fd-4704-a472-8965dd5d4230-konnectivity-ca\") pod \"konnectivity-agent-9glvk\" (UID: \"9f2fe74e-03fd-4704-a472-8965dd5d4230\") " pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287871 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bfcd6894-b47b-4ecd-9dc2-38235226086c-serviceca\") pod \"node-ca-rnnrw\" (UID: \"bfcd6894-b47b-4ecd-9dc2-38235226086c\") " pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.287961 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-sysctl-conf\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.288097 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d258f463-231f-41f9-89f2-e2077274343f-iptables-alerter-script\") pod \"iptables-alerter-jn24t\" (UID: \"d258f463-231f-41f9-89f2-e2077274343f\") " pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.288155 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-os-release\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.288156 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05818555-acf1-4dff-9440-e639a27413b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.288103 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d593cf7-aaaa-48f9-8f72-c823d8378efe-host-var-lib-kubelet\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.288358 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bfcd6894-b47b-4ecd-9dc2-38235226086c-serviceca\") pod \"node-ca-rnnrw\" (UID: \"bfcd6894-b47b-4ecd-9dc2-38235226086c\") " pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.289218 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2617985b-2645-4ad2-b29c-102d5b3acbd4-etc-tuned\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.289291 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f2fe74e-03fd-4704-a472-8965dd5d4230-agent-certs\") pod \"konnectivity-agent-9glvk\" (UID: \"9f2fe74e-03fd-4704-a472-8965dd5d4230\") " pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.289297 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f277f07-7a2c-4881-b1ba-aefff89d1279-ovn-node-metrics-cert\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.290530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.289542 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2617985b-2645-4ad2-b29c-102d5b3acbd4-tmp\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.297668 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.297646 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdsnm\" (UniqueName: \"kubernetes.io/projected/8d593cf7-aaaa-48f9-8f72-c823d8378efe-kube-api-access-mdsnm\") pod \"multus-sksph\" (UID: \"8d593cf7-aaaa-48f9-8f72-c823d8378efe\") " pod="openshift-multus/multus-sksph" Jul 07 17:54:16.299570 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.299244 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:16.299570 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.299271 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:16.299570 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.299286 2557 projected.go:196] Error preparing data for projected volume kube-api-access-vsgbp for pod openshift-network-diagnostics/network-check-target-dq27h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:16.299570 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.299365 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp podName:d5bc64ad-338f-4963-ad60-aedac86a2a61 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:16.799345698 +0000 UTC m=+2.052462279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vsgbp" (UniqueName: "kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp") pod "network-check-target-dq27h" (UID: "d5bc64ad-338f-4963-ad60-aedac86a2a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:16.301663 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.301637 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brw2\" (UniqueName: \"kubernetes.io/projected/d207b749-497b-4625-9510-3c0855f6b245-kube-api-access-8brw2\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:16.302090 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.302070 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qz7w\" (UniqueName: \"kubernetes.io/projected/d258f463-231f-41f9-89f2-e2077274343f-kube-api-access-2qz7w\") pod \"iptables-alerter-jn24t\" (UID: \"d258f463-231f-41f9-89f2-e2077274343f\") " pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.302177 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.302114 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpf96\" (UniqueName: \"kubernetes.io/projected/2617985b-2645-4ad2-b29c-102d5b3acbd4-kube-api-access-dpf96\") pod \"tuned-nsg2b\" (UID: \"2617985b-2645-4ad2-b29c-102d5b3acbd4\") " pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.302249 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.302232 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlfg\" (UniqueName: \"kubernetes.io/projected/05818555-acf1-4dff-9440-e639a27413b9-kube-api-access-qwlfg\") pod \"multus-additional-cni-plugins-h8npv\" (UID: \"05818555-acf1-4dff-9440-e639a27413b9\") " pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.302289 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.302269 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdmzj\" (UniqueName: \"kubernetes.io/projected/1fffc956-ac04-4a88-a353-a9aa99e3a8a3-kube-api-access-cdmzj\") pod \"node-resolver-52tkx\" (UID: \"1fffc956-ac04-4a88-a353-a9aa99e3a8a3\") " pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.302837 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.302816 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9jc\" (UniqueName: \"kubernetes.io/projected/bfcd6894-b47b-4ecd-9dc2-38235226086c-kube-api-access-cb9jc\") pod \"node-ca-rnnrw\" (UID: \"bfcd6894-b47b-4ecd-9dc2-38235226086c\") " pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.303250 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.303234 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffwtc\" (UniqueName: \"kubernetes.io/projected/a713dd27-8e40-4565-8c88-4378a67efc21-kube-api-access-ffwtc\") pod \"aws-ebs-csi-driver-node-6bz2d\" (UID: \"a713dd27-8e40-4565-8c88-4378a67efc21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.303311 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.303295 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkh9f\" (UniqueName: \"kubernetes.io/projected/4f277f07-7a2c-4881-b1ba-aefff89d1279-kube-api-access-wkh9f\") pod \"ovnkube-node-j7v76\" (UID: \"4f277f07-7a2c-4881-b1ba-aefff89d1279\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.384822 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.384745 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:16.413217 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.413180 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h8npv" Jul 07 17:54:16.429018 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.428995 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" Jul 07 17:54:16.436014 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.435990 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda713dd27_8e40_4565_8c88_4378a67efc21.slice/crio-7178bbcc7bec8223003fb15b6725a4dd258a9c20cd66828e68140174a50a0011 WatchSource:0}: Error finding container 7178bbcc7bec8223003fb15b6725a4dd258a9c20cd66828e68140174a50a0011: Status 404 returned error can't find the container with id 7178bbcc7bec8223003fb15b6725a4dd258a9c20cd66828e68140174a50a0011 Jul 07 17:54:16.449301 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.449280 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" Jul 07 17:54:16.456911 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.456881 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2617985b_2645_4ad2_b29c_102d5b3acbd4.slice/crio-e6585d11d5d1f8444516a3b55169c945cf7e953c535db71d0bee708f60cd92eb WatchSource:0}: Error finding container e6585d11d5d1f8444516a3b55169c945cf7e953c535db71d0bee708f60cd92eb: Status 404 returned error can't find the container with id e6585d11d5d1f8444516a3b55169c945cf7e953c535db71d0bee708f60cd92eb Jul 07 17:54:16.462050 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.462031 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:16.469589 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.469562 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2fe74e_03fd_4704_a472_8965dd5d4230.slice/crio-db570dc8e184e705cd324c6bc87ae4f04bf1910c7398590f998a0619614f7f9f WatchSource:0}: Error finding container db570dc8e184e705cd324c6bc87ae4f04bf1910c7398590f998a0619614f7f9f: Status 404 returned error can't find the container with id db570dc8e184e705cd324c6bc87ae4f04bf1910c7398590f998a0619614f7f9f Jul 07 17:54:16.471261 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.471241 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:16.478261 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.478231 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f277f07_7a2c_4881_b1ba_aefff89d1279.slice/crio-167e18949e46fdd5057e3f6db05204c0aec3896ccba209039bd55926fab68ef9 WatchSource:0}: Error finding container 167e18949e46fdd5057e3f6db05204c0aec3896ccba209039bd55926fab68ef9: Status 404 returned error can't find the container with id 167e18949e46fdd5057e3f6db05204c0aec3896ccba209039bd55926fab68ef9 Jul 07 17:54:16.478984 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.478969 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jn24t" Jul 07 17:54:16.489629 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.489606 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd258f463_231f_41f9_89f2_e2077274343f.slice/crio-28c06ecbffe6044ef1f372f8bc5f9ab54f57fb6cc08c661b57cb47140dc659e0 WatchSource:0}: Error finding container 28c06ecbffe6044ef1f372f8bc5f9ab54f57fb6cc08c661b57cb47140dc659e0: Status 404 returned error can't find the container with id 28c06ecbffe6044ef1f372f8bc5f9ab54f57fb6cc08c661b57cb47140dc659e0 Jul 07 17:54:16.492292 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.492268 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sksph" Jul 07 17:54:16.500944 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.500910 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d593cf7_aaaa_48f9_8f72_c823d8378efe.slice/crio-b6d8cf1347b8b39c354b5242d740b7ffe495223498ba86e6ab1476b50ca2b062 WatchSource:0}: Error finding container b6d8cf1347b8b39c354b5242d740b7ffe495223498ba86e6ab1476b50ca2b062: Status 404 returned error can't find the container with id b6d8cf1347b8b39c354b5242d740b7ffe495223498ba86e6ab1476b50ca2b062 Jul 07 17:54:16.507201 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.507178 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rnnrw" Jul 07 17:54:16.512184 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.512160 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52tkx" Jul 07 17:54:16.515468 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.515389 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcd6894_b47b_4ecd_9dc2_38235226086c.slice/crio-561446fff73fdc0c600c7da42dced8174bb038eb4f1b31d333dd513681a7c2c1 WatchSource:0}: Error finding container 561446fff73fdc0c600c7da42dced8174bb038eb4f1b31d333dd513681a7c2c1: Status 404 returned error can't find the container with id 561446fff73fdc0c600c7da42dced8174bb038eb4f1b31d333dd513681a7c2c1 Jul 07 17:54:16.520385 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:16.520364 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fffc956_ac04_4a88_a353_a9aa99e3a8a3.slice/crio-347abfb0f2ffa299e0011f65348671dc4beae06d4e812ed03efd346ba10996ab WatchSource:0}: Error finding container 347abfb0f2ffa299e0011f65348671dc4beae06d4e812ed03efd346ba10996ab: Status 404 returned error can't find the container with id 347abfb0f2ffa299e0011f65348671dc4beae06d4e812ed03efd346ba10996ab Jul 07 17:54:16.790867 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.790778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:16.790867 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.790826 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:16.791090 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.790998 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:16.791090 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.791059 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret podName:1cd8e48b-8053-42a2-b2fb-777e20a75ed1 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:17.791039689 +0000 UTC m=+3.044156251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret") pod "global-pull-secret-syncer-6s26g" (UID: "1cd8e48b-8053-42a2-b2fb-777e20a75ed1") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:16.791215 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.791133 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:16.791215 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.791169 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs podName:d207b749-497b-4625-9510-3c0855f6b245 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:17.791157363 +0000 UTC m=+3.044273922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs") pod "network-metrics-daemon-7zfpw" (UID: "d207b749-497b-4625-9510-3c0855f6b245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:16.891948 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:16.891910 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:16.892159 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.892126 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:16.892207 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.892164 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:16.892207 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.892181 2557 projected.go:196] Error preparing data for projected volume kube-api-access-vsgbp for pod openshift-network-diagnostics/network-check-target-dq27h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:16.892308 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:16.892289 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp podName:d5bc64ad-338f-4963-ad60-aedac86a2a61 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:17.892265981 +0000 UTC m=+3.145382568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsgbp" (UniqueName: "kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp") pod "network-check-target-dq27h" (UID: "d5bc64ad-338f-4963-ad60-aedac86a2a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:17.135964 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.135830 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:49:16 +0000 UTC" deadline="2027-12-27 12:11:58.51467567 +0000 UTC" Jul 07 17:54:17.135964 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.135905 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12906h17m41.378775623s" Jul 07 17:54:17.218511 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.218453 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:17.218709 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.218584 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:17.219050 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.219022 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:17.219168 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.219146 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:17.220801 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.219348 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:17.220801 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.219439 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:17.261625 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.261562 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8npv" event={"ID":"05818555-acf1-4dff-9440-e639a27413b9","Type":"ContainerStarted","Data":"02d18a06e35cdbb23e8a6b907a4647803fab24586e9444e13f45749e425e5b01"} Jul 07 17:54:17.280215 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.280070 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sksph" event={"ID":"8d593cf7-aaaa-48f9-8f72-c823d8378efe","Type":"ContainerStarted","Data":"b6d8cf1347b8b39c354b5242d740b7ffe495223498ba86e6ab1476b50ca2b062"} Jul 07 17:54:17.294378 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.294253 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerStarted","Data":"167e18949e46fdd5057e3f6db05204c0aec3896ccba209039bd55926fab68ef9"} Jul 07 17:54:17.304021 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.303951 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9glvk" event={"ID":"9f2fe74e-03fd-4704-a472-8965dd5d4230","Type":"ContainerStarted","Data":"db570dc8e184e705cd324c6bc87ae4f04bf1910c7398590f998a0619614f7f9f"} Jul 07 17:54:17.322647 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.322611 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" event={"ID":"a713dd27-8e40-4565-8c88-4378a67efc21","Type":"ContainerStarted","Data":"7178bbcc7bec8223003fb15b6725a4dd258a9c20cd66828e68140174a50a0011"} Jul 07 17:54:17.325469 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.325439 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" event={"ID":"79639d5bc5ecaca9c381236434449ff7","Type":"ContainerStarted","Data":"f481044dbb7bf8447ab0f185915fe3a895d082b6b4a6c4d13a0fdd8a216e133f"} Jul 07 17:54:17.332661 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.332595 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"ca7e154a214d139ef80f0ff105532d9a","Type":"ContainerStarted","Data":"42004c8c60437a6f64e0a63e66f4e13be3116705c4b9ff11dde5694daee66190"} Jul 07 17:54:17.351462 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.351429 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52tkx" event={"ID":"1fffc956-ac04-4a88-a353-a9aa99e3a8a3","Type":"ContainerStarted","Data":"347abfb0f2ffa299e0011f65348671dc4beae06d4e812ed03efd346ba10996ab"} Jul 07 17:54:17.379363 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.379323 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rnnrw" event={"ID":"bfcd6894-b47b-4ecd-9dc2-38235226086c","Type":"ContainerStarted","Data":"561446fff73fdc0c600c7da42dced8174bb038eb4f1b31d333dd513681a7c2c1"} Jul 07 17:54:17.407353 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.407295 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jn24t" event={"ID":"d258f463-231f-41f9-89f2-e2077274343f","Type":"ContainerStarted","Data":"28c06ecbffe6044ef1f372f8bc5f9ab54f57fb6cc08c661b57cb47140dc659e0"} Jul 07 17:54:17.426006 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.425964 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" event={"ID":"2617985b-2645-4ad2-b29c-102d5b3acbd4","Type":"ContainerStarted","Data":"e6585d11d5d1f8444516a3b55169c945cf7e953c535db71d0bee708f60cd92eb"} Jul 07 17:54:17.618793 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.618567 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:17.802199 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.802105 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:17.802199 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.802167 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:17.802459 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.802294 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:17.802459 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.802294 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:17.802459 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.802381 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret podName:1cd8e48b-8053-42a2-b2fb-777e20a75ed1 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:19.802342505 +0000 UTC m=+5.055459069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret") pod "global-pull-secret-syncer-6s26g" (UID: "1cd8e48b-8053-42a2-b2fb-777e20a75ed1") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:17.802459 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.802402 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs podName:d207b749-497b-4625-9510-3c0855f6b245 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:19.802390789 +0000 UTC m=+5.055507356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs") pod "network-metrics-daemon-7zfpw" (UID: "d207b749-497b-4625-9510-3c0855f6b245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:17.903912 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:17.903823 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:17.904126 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.903996 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:17.904126 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.904020 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:17.904126 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.904034 2557 projected.go:196] Error preparing data for projected volume kube-api-access-vsgbp for pod openshift-network-diagnostics/network-check-target-dq27h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:17.904126 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:17.904096 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp podName:d5bc64ad-338f-4963-ad60-aedac86a2a61 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:19.904074452 +0000 UTC m=+5.157191013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsgbp" (UniqueName: "kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp") pod "network-check-target-dq27h" (UID: "d5bc64ad-338f-4963-ad60-aedac86a2a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:18.136220 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:18.136123 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:49:16 +0000 UTC" deadline="2028-02-23 12:00:21.320672696 +0000 UTC" Jul 07 17:54:18.136220 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:18.136170 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14298h6m3.184507939s" Jul 07 17:54:18.710855 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:18.710808 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:19.020831 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:19.020267 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:19.218548 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:19.218516 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:19.219074 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:19.218523 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:19.219074 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.218707 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:19.219074 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.218819 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:19.219074 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:19.218871 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:19.219074 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.218946 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:19.822129 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:19.822086 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:19.822336 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:19.822148 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:19.822336 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.822322 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:19.822503 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.822404 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret podName:1cd8e48b-8053-42a2-b2fb-777e20a75ed1 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:23.822383849 +0000 UTC m=+9.075500425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret") pod "global-pull-secret-syncer-6s26g" (UID: "1cd8e48b-8053-42a2-b2fb-777e20a75ed1") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:19.822869 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.822851 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:19.822944 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.822906 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs podName:d207b749-497b-4625-9510-3c0855f6b245 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:23.822889611 +0000 UTC m=+9.076006180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs") pod "network-metrics-daemon-7zfpw" (UID: "d207b749-497b-4625-9510-3c0855f6b245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:19.923165 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:19.922792 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:19.923165 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.923025 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:19.923165 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.923050 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:19.923165 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.923063 2557 projected.go:196] Error preparing data for projected volume kube-api-access-vsgbp for pod openshift-network-diagnostics/network-check-target-dq27h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:19.923165 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:19.923125 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp podName:d5bc64ad-338f-4963-ad60-aedac86a2a61 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:23.923105026 +0000 UTC m=+9.176221589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsgbp" (UniqueName: "kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp") pod "network-check-target-dq27h" (UID: "d5bc64ad-338f-4963-ad60-aedac86a2a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:21.220503 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:21.220458 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:21.221022 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:21.220468 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:21.221022 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:21.220592 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:21.221022 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:21.220641 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:21.221022 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:21.220733 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:21.221022 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:21.220802 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:23.219581 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:23.219548 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:23.219995 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:23.219589 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:23.219995 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.219701 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:23.219995 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:23.219756 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:23.219995 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.219848 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:23.219995 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.219916 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:23.858675 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:23.858631 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:23.858871 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:23.858704 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:23.858871 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.858794 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:23.858871 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.858862 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:23.859065 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.858894 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs podName:d207b749-497b-4625-9510-3c0855f6b245 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:31.858871791 +0000 UTC m=+17.111988366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs") pod "network-metrics-daemon-7zfpw" (UID: "d207b749-497b-4625-9510-3c0855f6b245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:23.859065 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.858920 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret podName:1cd8e48b-8053-42a2-b2fb-777e20a75ed1 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:31.858903341 +0000 UTC m=+17.112019907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret") pod "global-pull-secret-syncer-6s26g" (UID: "1cd8e48b-8053-42a2-b2fb-777e20a75ed1") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:23.959808 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:23.959765 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:23.960002 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.959972 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:23.960002 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.960000 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:23.960089 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.960015 2557 projected.go:196] Error preparing data for projected volume kube-api-access-vsgbp for pod openshift-network-diagnostics/network-check-target-dq27h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:23.960089 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:23.960081 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp podName:d5bc64ad-338f-4963-ad60-aedac86a2a61 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:31.960060832 +0000 UTC m=+17.213177395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsgbp" (UniqueName: "kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp") pod "network-check-target-dq27h" (UID: "d5bc64ad-338f-4963-ad60-aedac86a2a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:25.220361 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:25.219420 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:25.220361 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:25.219532 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:25.220361 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:25.220167 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:25.220361 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:25.220268 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:25.220361 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:25.220365 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:25.221024 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:25.220451 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:27.218471 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:27.218196 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:27.218961 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:27.218203 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:27.218961 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:27.218577 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:27.218961 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:27.218253 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:27.218961 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:27.218724 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:27.218961 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:27.218840 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:29.217939 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:29.217907 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:29.217939 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:29.217926 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:29.218429 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:29.217907 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:29.218429 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:29.218041 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:29.218429 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:29.218096 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:29.218429 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:29.218162 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:31.218708 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:31.218646 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:31.218708 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:31.218679 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:31.218708 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:31.218709 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:31.219254 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:31.218852 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:31.219254 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:31.218974 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:31.219254 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:31.219045 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:31.934115 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:31.934074 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:31.934115 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:31.934116 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:31.934402 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:31.934223 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:31.934402 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:31.934224 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:31.934402 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:31.934274 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret podName:1cd8e48b-8053-42a2-b2fb-777e20a75ed1 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:47.934260097 +0000 UTC m=+33.187376661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret") pod "global-pull-secret-syncer-6s26g" (UID: "1cd8e48b-8053-42a2-b2fb-777e20a75ed1") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:31.934402 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:31.934287 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs podName:d207b749-497b-4625-9510-3c0855f6b245 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:47.934280889 +0000 UTC m=+33.187397450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs") pod "network-metrics-daemon-7zfpw" (UID: "d207b749-497b-4625-9510-3c0855f6b245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:32.035192 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:32.035156 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:32.035361 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:32.035341 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:32.035411 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:32.035367 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:32.035411 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:32.035377 2557 projected.go:196] Error preparing data for projected volume kube-api-access-vsgbp for pod openshift-network-diagnostics/network-check-target-dq27h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:32.035479 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:32.035429 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp podName:d5bc64ad-338f-4963-ad60-aedac86a2a61 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:48.035415024 +0000 UTC m=+33.288531581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsgbp" (UniqueName: "kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp") pod "network-check-target-dq27h" (UID: "d5bc64ad-338f-4963-ad60-aedac86a2a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:33.218535 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:33.218007 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:33.218535 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:33.218140 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:33.218535 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:33.218017 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:33.218535 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:33.218180 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:33.218535 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:33.218352 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:33.218535 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:33.218468 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:35.219125 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:35.219089 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:35.219597 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:35.219189 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:35.219597 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:35.219213 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:35.219597 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:35.219307 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:35.219597 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:35.219371 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:35.219597 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:35.219491 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:37.218350 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:37.218293 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:37.218901 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:37.218463 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:37.218901 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:37.218293 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:37.218901 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:37.218881 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:37.219112 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:37.219089 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:37.219218 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:37.219187 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:38.479752 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.479086 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" event={"ID":"2617985b-2645-4ad2-b29c-102d5b3acbd4","Type":"ContainerStarted","Data":"8f954e420a74832940d580302a38def815acecc868fd72c4640e2081128eb6df"} Jul 07 17:54:38.482279 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.482253 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sksph" event={"ID":"8d593cf7-aaaa-48f9-8f72-c823d8378efe","Type":"ContainerStarted","Data":"3c90934c783bbeff6fae3c4ca93b7f3f68e136c487d3804aba0d4fb464c68e98"} Jul 07 17:54:38.485590 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.485572 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 17:54:38.485960 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.485929 2557 generic.go:356] "Generic (PLEG): container finished" podID="4f277f07-7a2c-4881-b1ba-aefff89d1279" containerID="4a2a0fe619a196cfa5dff5dc6325b697f893cf7d4be793c0e6f86eec94e87ee8" exitCode=1 Jul 07 17:54:38.486035 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.485975 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerStarted","Data":"ba365d89ee00f9902d9a87d5ac14dc6393ea884d45f70c87502d6954f508361d"} Jul 07 17:54:38.486078 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.486053 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerStarted","Data":"212b529165d4fd17a483c9e436e2d1698c1746ed0f4fb82ec247fb986f156769"} Jul 07 17:54:38.486131 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.486107 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerStarted","Data":"81f42d405d685782f852888486d9dbec3b87d36515446676483d44dedad998e4"} Jul 07 17:54:38.486180 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.486154 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerStarted","Data":"9f94941eee112a9e9eb4ca79ec62f93f6a8fadafd6d08d4e4be504e6b0e03233"} Jul 07 17:54:38.486235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.486201 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerDied","Data":"4a2a0fe619a196cfa5dff5dc6325b697f893cf7d4be793c0e6f86eec94e87ee8"} Jul 07 17:54:38.486235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.486232 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerStarted","Data":"ed254645f871713fcf835bef1640f796969a0dd26ac10aba2beb0628cb5f8456"} Jul 07 17:54:38.487689 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.487668 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" event={"ID":"79639d5bc5ecaca9c381236434449ff7","Type":"ContainerStarted","Data":"e3150c5ab35ef1e46b1359b58d3d951b0ffa84b7d2a14e6b30c9ce877bcdec30"} Jul 07 17:54:38.498892 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.498853 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nsg2b" podStartSLOduration=2.29865232 podStartE2EDuration="23.498841072s" totalImagesPullingTime="21.200188752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:38.498369979 +0000 UTC m=+23.751486580" watchObservedRunningTime="2026-07-07 17:54:38.498841072 +0000 UTC m=+23.751957651" Jul 07 17:54:38.519805 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:38.519763 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sksph" podStartSLOduration=2.363992162 podStartE2EDuration="23.51974959s" totalImagesPullingTime="21.155757428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:38.519552126 +0000 UTC m=+23.772668745" watchObservedRunningTime="2026-07-07 17:54:38.51974959 +0000 UTC m=+23.772866170" Jul 07 17:54:39.218444 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.218182 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:39.218584 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.218182 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:39.218584 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:39.218485 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:39.218584 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.218182 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:39.218584 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:39.218556 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:39.218711 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:39.218626 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:39.491300 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.491177 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52tkx" event={"ID":"1fffc956-ac04-4a88-a353-a9aa99e3a8a3","Type":"ContainerStarted","Data":"031a71c5e4dcd086d39af6590867521b8a711738a1563644c1139115f9df8596"} Jul 07 17:54:39.492435 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.492411 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rnnrw" event={"ID":"bfcd6894-b47b-4ecd-9dc2-38235226086c","Type":"ContainerStarted","Data":"bbc5ed45872e6ce7d4794bb12b312233e1c2069556187b46e4f1d0d692ca5768"} Jul 07 17:54:39.493737 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.493717 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jn24t" event={"ID":"d258f463-231f-41f9-89f2-e2077274343f","Type":"ContainerStarted","Data":"11e0766266c132f2a02096209d37ee38d5dc26b3a8662227f13422d134661fa5"} Jul 07 17:54:39.495110 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.495081 2557 generic.go:356] "Generic (PLEG): container finished" podID="05818555-acf1-4dff-9440-e639a27413b9" containerID="62a0b126de1acc65a3ee97045a25c6ed10b146fce631c66481c30e471b1626ea" exitCode=0 Jul 07 17:54:39.495191 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.495161 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8npv" event={"ID":"05818555-acf1-4dff-9440-e639a27413b9","Type":"ContainerDied","Data":"62a0b126de1acc65a3ee97045a25c6ed10b146fce631c66481c30e471b1626ea"} Jul 07 17:54:39.496660 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.496586 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9glvk" event={"ID":"9f2fe74e-03fd-4704-a472-8965dd5d4230","Type":"ContainerStarted","Data":"f85cc1682154d6fddc15e072a41228a4d823e6c803b7ca4add2f34096406c4fb"} Jul 07 17:54:39.497931 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.497909 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" event={"ID":"a713dd27-8e40-4565-8c88-4378a67efc21","Type":"ContainerStarted","Data":"f22671f652e36c522d2a807f18ddf27e7c9ecfa01b1ad216d04c93a97eebe1b3"} Jul 07 17:54:39.499297 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.499278 2557 generic.go:356] "Generic (PLEG): container finished" podID="ca7e154a214d139ef80f0ff105532d9a" containerID="6c3b440ebe0bb2cbe9c9a12041dd5c1a05830d28fd6dae1123589baeea127fcf" exitCode=0 Jul 07 17:54:39.499402 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.499369 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"ca7e154a214d139ef80f0ff105532d9a","Type":"ContainerDied","Data":"6c3b440ebe0bb2cbe9c9a12041dd5c1a05830d28fd6dae1123589baeea127fcf"} Jul 07 17:54:39.508491 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.508458 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" podStartSLOduration=23.508446217 podStartE2EDuration="23.508446217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:38.535144675 +0000 UTC m=+23.788261273" watchObservedRunningTime="2026-07-07 17:54:39.508446217 +0000 UTC m=+24.761562808" Jul 07 17:54:39.508799 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.508776 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-52tkx" podStartSLOduration=2.371366611 podStartE2EDuration="23.508770132s" totalImagesPullingTime="21.137403521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:39.508627499 +0000 UTC m=+24.761744079" watchObservedRunningTime="2026-07-07 17:54:39.508770132 +0000 UTC m=+24.761886711" Jul 07 17:54:39.541413 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.541373 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rnnrw" podStartSLOduration=2.420810177 podStartE2EDuration="23.541355178s" totalImagesPullingTime="21.120545001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:39.540113668 +0000 UTC m=+24.793230268" watchObservedRunningTime="2026-07-07 17:54:39.541355178 +0000 UTC m=+24.794471759" Jul 07 17:54:39.596309 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.596259 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9glvk" podStartSLOduration=3.4293875099999998 podStartE2EDuration="24.596244621s" totalImagesPullingTime="21.166857111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:39.596061367 +0000 UTC m=+24.849177947" watchObservedRunningTime="2026-07-07 17:54:39.596244621 +0000 UTC m=+24.849361200" Jul 07 17:54:39.611861 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:39.611807 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jn24t" podStartSLOduration=3.731177933 podStartE2EDuration="24.611789267s" totalImagesPullingTime="20.880611334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:39.611242659 +0000 UTC m=+24.864359257" watchObservedRunningTime="2026-07-07 17:54:39.611789267 +0000 UTC m=+24.864905848" Jul 07 17:54:40.427464 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:40.427443 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 17:54:40.503543 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:40.503513 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" event={"ID":"a713dd27-8e40-4565-8c88-4378a67efc21","Type":"ContainerStarted","Data":"ca2108698ffca374ca82ff0246214c64eee56f646bbe4340e1b697867c3594c0"} Jul 07 17:54:40.505546 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:40.505516 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"ca7e154a214d139ef80f0ff105532d9a","Type":"ContainerStarted","Data":"0cefc57de9c75330690c2be7a486755f9d9fd1f304324a7116617da05671b1ba"} Jul 07 17:54:40.532646 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:40.532574 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" podStartSLOduration=24.53256161 podStartE2EDuration="24.53256161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:40.532440343 +0000 UTC m=+25.785556922" watchObservedRunningTime="2026-07-07 17:54:40.53256161 +0000 UTC m=+25.785678189" Jul 07 17:54:41.153402 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:41.153279 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T17:54:40.427459818Z","UUID":"51a5d317-342f-403e-8fdd-408eac2480cc","Handler":null,"Name":"","Endpoint":""} Jul 07 17:54:41.154886 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:41.154867 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 17:54:41.154994 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:41.154902 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 17:54:41.217956 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:41.217924 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:41.218129 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:41.218041 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:41.218196 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:41.218127 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:41.218196 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:41.218034 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:41.218196 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:41.218164 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:41.218372 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:41.218237 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:41.511255 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:41.511168 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 17:54:41.511801 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:41.511614 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerStarted","Data":"b9ad2ce764dee0dec8a2cbeb3a81076de5edf8986554d255ef5c3dafe6f7e0ad"} Jul 07 17:54:42.392615 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:42.392579 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:42.393215 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:42.393188 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:42.516696 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:42.516613 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" event={"ID":"a713dd27-8e40-4565-8c88-4378a67efc21","Type":"ContainerStarted","Data":"7a536a42626a475e7c1f997ede8af4abbef41b850ec1f5ccc172f0b500725cf2"} Jul 07 17:54:42.517165 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:42.517147 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:42.518130 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:42.518111 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9glvk" Jul 07 17:54:42.538294 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:42.538230 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bz2d" podStartSLOduration=3.309208012 podStartE2EDuration="27.538210645s" totalImagesPullingTime="24.229002633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:42.537381334 +0000 UTC m=+27.790497934" watchObservedRunningTime="2026-07-07 17:54:42.538210645 +0000 UTC m=+27.791327227" Jul 07 17:54:43.217815 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:43.217734 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:43.217815 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:43.217770 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:43.218064 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:43.217738 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:43.218064 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:43.217877 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:43.218064 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:43.217950 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:43.218064 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:43.218040 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:43.523366 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:43.523279 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 17:54:43.523997 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:43.523745 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerStarted","Data":"bae520f288515c7592590e749b3255aac5bf87904ca8dabcc720dc0c59f0be9c"} Jul 07 17:54:43.524150 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:43.524130 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:43.524371 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:43.524352 2557 scope.go:117] "RemoveContainer" containerID="4a2a0fe619a196cfa5dff5dc6325b697f893cf7d4be793c0e6f86eec94e87ee8" Jul 07 17:54:43.547258 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:43.547233 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:44.531133 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.530935 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 17:54:44.531704 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.531594 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" event={"ID":"4f277f07-7a2c-4881-b1ba-aefff89d1279","Type":"ContainerStarted","Data":"6a75b3729a0d6bbc0bd44c8d2259bfcb76687a354bf4f7130d6517cdd1b79630"} Jul 07 17:54:44.531858 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.531842 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:44.531939 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.531915 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:44.549873 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.549849 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:54:44.573403 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.573350 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" podStartSLOduration=8.231387965 podStartE2EDuration="29.573316489s" totalImagesPullingTime="21.341928524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:44.573160287 +0000 UTC m=+29.826276891" watchObservedRunningTime="2026-07-07 17:54:44.573316489 +0000 UTC m=+29.826433069" Jul 07 17:54:44.638433 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.638402 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7zfpw"] Jul 07 17:54:44.638594 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.638523 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:44.638668 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:44.638647 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:44.641356 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.641310 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dq27h"] Jul 07 17:54:44.641499 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.641442 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:44.641565 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:44.641540 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:44.642026 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.641998 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6s26g"] Jul 07 17:54:44.642117 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:44.642093 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:44.642190 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:44.642174 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:46.217929 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:46.217901 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:46.218264 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:46.217932 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:46.218264 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:46.217937 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:46.218264 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:46.218010 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:46.218264 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:46.218085 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:46.218264 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:46.218154 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:46.538200 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:46.538111 2557 generic.go:356] "Generic (PLEG): container finished" podID="05818555-acf1-4dff-9440-e639a27413b9" containerID="56d83add4ab6132abef8d428390a900316974fc70c1354e6a655e49520ef2ff1" exitCode=0 Jul 07 17:54:46.538200 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:46.538190 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8npv" event={"ID":"05818555-acf1-4dff-9440-e639a27413b9","Type":"ContainerDied","Data":"56d83add4ab6132abef8d428390a900316974fc70c1354e6a655e49520ef2ff1"} Jul 07 17:54:47.955436 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:47.955396 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:47.955436 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:47.955438 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:47.955856 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:47.955547 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:47.955856 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:47.955548 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:47.955856 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:47.955601 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret podName:1cd8e48b-8053-42a2-b2fb-777e20a75ed1 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:19.955587158 +0000 UTC m=+65.208703720 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret") pod "global-pull-secret-syncer-6s26g" (UID: "1cd8e48b-8053-42a2-b2fb-777e20a75ed1") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:47.955856 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:47.955618 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs podName:d207b749-497b-4625-9510-3c0855f6b245 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:19.9556101 +0000 UTC m=+65.208726665 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs") pod "network-metrics-daemon-7zfpw" (UID: "d207b749-497b-4625-9510-3c0855f6b245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:48.055904 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:48.055863 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:48.056075 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:48.056036 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:48.056075 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:48.056058 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:48.056075 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:48.056068 2557 projected.go:196] Error preparing data for projected volume kube-api-access-vsgbp for pod openshift-network-diagnostics/network-check-target-dq27h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:48.056171 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:48.056125 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp podName:d5bc64ad-338f-4963-ad60-aedac86a2a61 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:20.056108488 +0000 UTC m=+65.309225046 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsgbp" (UniqueName: "kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp") pod "network-check-target-dq27h" (UID: "d5bc64ad-338f-4963-ad60-aedac86a2a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:48.218310 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:48.218223 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:48.218492 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:48.218310 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:48.218492 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:48.218313 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:48.218492 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:48.218433 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:48.218492 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:48.218492 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:48.218701 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:48.218585 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:48.546515 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:48.546430 2557 generic.go:356] "Generic (PLEG): container finished" podID="05818555-acf1-4dff-9440-e639a27413b9" containerID="87be021bab8a4c903e73e4c7d687ab051c2124e41ce4bb3c712b446ae2c001b0" exitCode=0 Jul 07 17:54:48.546515 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:48.546494 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8npv" event={"ID":"05818555-acf1-4dff-9440-e639a27413b9","Type":"ContainerDied","Data":"87be021bab8a4c903e73e4c7d687ab051c2124e41ce4bb3c712b446ae2c001b0"} Jul 07 17:54:50.218340 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.218148 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:50.218709 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.218191 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:50.218709 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:50.218442 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6s26g" podUID="1cd8e48b-8053-42a2-b2fb-777e20a75ed1" Jul 07 17:54:50.218709 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.218203 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:50.218709 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:50.218507 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dq27h" podUID="d5bc64ad-338f-4963-ad60-aedac86a2a61" Jul 07 17:54:50.218709 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:50.218583 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zfpw" podUID="d207b749-497b-4625-9510-3c0855f6b245" Jul 07 17:54:50.541990 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.541908 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeReady" Jul 07 17:54:50.542149 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.542058 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 17:54:50.554025 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.553997 2557 generic.go:356] "Generic (PLEG): container finished" podID="05818555-acf1-4dff-9440-e639a27413b9" containerID="dcc7ce4b5e1ee32dd2046b31b01234642731907e407b53bfbb1186403662211d" exitCode=0 Jul 07 17:54:50.554162 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.554035 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8npv" event={"ID":"05818555-acf1-4dff-9440-e639a27413b9","Type":"ContainerDied","Data":"dcc7ce4b5e1ee32dd2046b31b01234642731907e407b53bfbb1186403662211d"} Jul 07 17:54:50.597183 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.597153 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xd72b"] Jul 07 17:54:50.617336 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.617289 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h8zpz"] Jul 07 17:54:50.617466 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.617407 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.620777 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.620758 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dtpjm\"" Jul 07 17:54:50.620875 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.620777 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 17:54:50.622926 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.622907 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 17:54:50.631345 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.631308 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xd72b"] Jul 07 17:54:50.631423 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.631362 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h8zpz"] Jul 07 17:54:50.631460 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.631425 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h8zpz" Jul 07 17:54:50.634375 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.634357 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 17:54:50.634474 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.634377 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f6qmm\"" Jul 07 17:54:50.634474 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.634377 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.634474 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.634382 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.662204 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.662181 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt"] Jul 07 17:54:50.675374 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.675354 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:50.680394 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.680376 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.681003 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.680868 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sqwtn\"" Jul 07 17:54:50.681003 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.680882 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 07 17:54:50.681003 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.680882 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.681003 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.680949 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 07 17:54:50.683036 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.683017 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt"] Jul 07 17:54:50.776684 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.776654 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:50.776684 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.776687 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slk8l\" (UniqueName: \"kubernetes.io/projected/62bcc290-f56a-4710-b664-c6234385c9a2-kube-api-access-slk8l\") pod \"ingress-canary-h8zpz\" (UID: \"62bcc290-f56a-4710-b664-c6234385c9a2\") " pod="openshift-ingress-canary/ingress-canary-h8zpz" Jul 07 17:54:50.776890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.776717 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62bcc290-f56a-4710-b664-c6234385c9a2-cert\") pod \"ingress-canary-h8zpz\" (UID: \"62bcc290-f56a-4710-b664-c6234385c9a2\") " pod="openshift-ingress-canary/ingress-canary-h8zpz" Jul 07 17:54:50.776890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.776755 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:50.776890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.776777 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wdsw\" (UniqueName: \"kubernetes.io/projected/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-kube-api-access-9wdsw\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:50.776890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.776818 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrp2h\" (UniqueName: \"kubernetes.io/projected/29ef67eb-8543-4716-8f68-fe90e8cbe171-kube-api-access-qrp2h\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.776890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.776847 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29ef67eb-8543-4716-8f68-fe90e8cbe171-config-volume\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.776890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.776862 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.776890 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.776883 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ef67eb-8543-4716-8f68-fe90e8cbe171-tmp-dir\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.788262 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.788224 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc"] Jul 07 17:54:50.805390 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.805315 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-x8khq"] Jul 07 17:54:50.805480 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.805437 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:50.810610 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.810583 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 07 17:54:50.810610 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.810604 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 07 17:54:50.810774 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.810617 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.810774 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.810604 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.810774 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.810729 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vdm7c\"" Jul 07 17:54:50.819289 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.819269 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf"] Jul 07 17:54:50.819405 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.819396 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:50.829509 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.827044 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.829509 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.827164 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.829509 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.827409 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 17:54:50.829509 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.827569 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 17:54:50.829509 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.827672 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mrh48\"" Jul 07 17:54:50.834168 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.834146 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 17:54:50.835240 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.835222 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf"] Jul 07 17:54:50.835309 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.835252 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-x8khq"] Jul 07 17:54:50.835309 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.835266 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc"] Jul 07 17:54:50.835399 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.835370 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:50.837716 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.837698 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 17:54:50.837800 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.837716 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gp7cm\"" Jul 07 17:54:50.837800 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.837744 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.838130 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.838114 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.877625 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.877602 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:50.877780 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.877631 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slk8l\" (UniqueName: \"kubernetes.io/projected/62bcc290-f56a-4710-b664-c6234385c9a2-kube-api-access-slk8l\") pod \"ingress-canary-h8zpz\" (UID: \"62bcc290-f56a-4710-b664-c6234385c9a2\") " pod="openshift-ingress-canary/ingress-canary-h8zpz" Jul 07 17:54:50.877780 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.877661 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62bcc290-f56a-4710-b664-c6234385c9a2-cert\") pod \"ingress-canary-h8zpz\" (UID: \"62bcc290-f56a-4710-b664-c6234385c9a2\") " pod="openshift-ingress-canary/ingress-canary-h8zpz" Jul 07 17:54:50.877780 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.877681 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:50.877780 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.877698 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wdsw\" (UniqueName: \"kubernetes.io/projected/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-kube-api-access-9wdsw\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:50.877780 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:50.877768 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:50.878031 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.877796 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrp2h\" (UniqueName: \"kubernetes.io/projected/29ef67eb-8543-4716-8f68-fe90e8cbe171-kube-api-access-qrp2h\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.878031 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:50.877844 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls podName:c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:51.377824432 +0000 UTC m=+36.630941003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xlgtt" (UID: "c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:50.878031 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.877868 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29ef67eb-8543-4716-8f68-fe90e8cbe171-config-volume\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.878031 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.877898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.878031 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.877930 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ef67eb-8543-4716-8f68-fe90e8cbe171-tmp-dir\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.878031 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:50.877983 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:50.878387 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:50.878060 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls podName:29ef67eb-8543-4716-8f68-fe90e8cbe171 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:51.378043575 +0000 UTC m=+36.631160135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls") pod "dns-default-xd72b" (UID: "29ef67eb-8543-4716-8f68-fe90e8cbe171") : secret "dns-default-metrics-tls" not found Jul 07 17:54:50.878387 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.878251 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ef67eb-8543-4716-8f68-fe90e8cbe171-tmp-dir\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.878509 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.878489 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29ef67eb-8543-4716-8f68-fe90e8cbe171-config-volume\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.878569 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.878550 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:50.882460 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.882441 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62bcc290-f56a-4710-b664-c6234385c9a2-cert\") pod \"ingress-canary-h8zpz\" (UID: \"62bcc290-f56a-4710-b664-c6234385c9a2\") " pod="openshift-ingress-canary/ingress-canary-h8zpz" Jul 07 17:54:50.919538 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.919516 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wdsw\" (UniqueName: \"kubernetes.io/projected/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-kube-api-access-9wdsw\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:50.921114 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.921084 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrp2h\" (UniqueName: \"kubernetes.io/projected/29ef67eb-8543-4716-8f68-fe90e8cbe171-kube-api-access-qrp2h\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:50.921442 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.921424 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slk8l\" (UniqueName: \"kubernetes.io/projected/62bcc290-f56a-4710-b664-c6234385c9a2-kube-api-access-slk8l\") pod \"ingress-canary-h8zpz\" (UID: \"62bcc290-f56a-4710-b664-c6234385c9a2\") " pod="openshift-ingress-canary/ingress-canary-h8zpz" Jul 07 17:54:50.939695 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.939676 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h8zpz" Jul 07 17:54:50.971803 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.971779 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng"] Jul 07 17:54:50.978468 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978449 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:50.978543 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978490 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlwpj\" (UniqueName: \"kubernetes.io/projected/68a85369-16d8-457b-a852-1bd45827378e-kube-api-access-qlwpj\") pod \"service-ca-operator-5d46d9ffd4-psftc\" (UID: \"68a85369-16d8-457b-a852-1bd45827378e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:50.978543 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978512 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:50.978543 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68a85369-16d8-457b-a852-1bd45827378e-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-psftc\" (UID: \"68a85369-16d8-457b-a852-1bd45827378e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:50.978645 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978574 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-snapshots\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:50.978645 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978626 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklpc\" (UniqueName: \"kubernetes.io/projected/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-kube-api-access-hklpc\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:50.978720 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978704 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-serving-cert\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:50.978757 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978740 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:50.978832 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978816 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68a85369-16d8-457b-a852-1bd45827378e-config\") pod \"service-ca-operator-5d46d9ffd4-psftc\" (UID: \"68a85369-16d8-457b-a852-1bd45827378e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:50.978891 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978874 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-tmp\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:50.978939 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.978900 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv9k2\" (UniqueName: \"kubernetes.io/projected/88ce489b-b618-41d1-acbe-6fc6423af618-kube-api-access-xv9k2\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:50.990106 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.990084 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng"] Jul 07 17:54:50.990200 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.990172 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng" Jul 07 17:54:50.994855 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.994827 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.995196 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.995179 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.995441 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:50.995428 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mdvlq\"" Jul 07 17:54:51.080054 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68a85369-16d8-457b-a852-1bd45827378e-config\") pod \"service-ca-operator-5d46d9ffd4-psftc\" (UID: \"68a85369-16d8-457b-a852-1bd45827378e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:51.080192 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080058 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-tmp\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.080192 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080086 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv9k2\" (UniqueName: \"kubernetes.io/projected/88ce489b-b618-41d1-acbe-6fc6423af618-kube-api-access-xv9k2\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:51.080192 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080177 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.080346 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080225 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlwpj\" (UniqueName: \"kubernetes.io/projected/68a85369-16d8-457b-a852-1bd45827378e-kube-api-access-qlwpj\") pod \"service-ca-operator-5d46d9ffd4-psftc\" (UID: \"68a85369-16d8-457b-a852-1bd45827378e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:51.080346 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080265 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.080346 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080294 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68a85369-16d8-457b-a852-1bd45827378e-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-psftc\" (UID: \"68a85369-16d8-457b-a852-1bd45827378e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:51.080504 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080372 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-snapshots\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.080504 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080399 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklpc\" (UniqueName: \"kubernetes.io/projected/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-kube-api-access-hklpc\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.080504 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080438 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-serving-cert\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.080504 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080465 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:51.080668 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.080503 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnkzv\" (UniqueName: \"kubernetes.io/projected/ababb634-f89d-41e2-aafb-f1d865334371-kube-api-access-gnkzv\") pod \"network-check-source-989f8b8dc-tz7ng\" (UID: \"ababb634-f89d-41e2-aafb-f1d865334371\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng" Jul 07 17:54:51.081200 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.081173 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68a85369-16d8-457b-a852-1bd45827378e-config\") pod \"service-ca-operator-5d46d9ffd4-psftc\" (UID: \"68a85369-16d8-457b-a852-1bd45827378e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:51.081614 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.081592 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-tmp\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.083232 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:51.083208 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:54:51.083392 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:51.083273 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls podName:88ce489b-b618-41d1-acbe-6fc6423af618 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:51.583256623 +0000 UTC m=+36.836373184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2fkvf" (UID: "88ce489b-b618-41d1-acbe-6fc6423af618") : secret "samples-operator-tls" not found Jul 07 17:54:51.085210 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.084930 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.085210 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.085156 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-snapshots\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.085954 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.085912 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.088983 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.088934 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68a85369-16d8-457b-a852-1bd45827378e-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-psftc\" (UID: \"68a85369-16d8-457b-a852-1bd45827378e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:51.091314 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.091284 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-serving-cert\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.095981 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.095957 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv9k2\" (UniqueName: \"kubernetes.io/projected/88ce489b-b618-41d1-acbe-6fc6423af618-kube-api-access-xv9k2\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:51.096056 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.096005 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlwpj\" (UniqueName: \"kubernetes.io/projected/68a85369-16d8-457b-a852-1bd45827378e-kube-api-access-qlwpj\") pod \"service-ca-operator-5d46d9ffd4-psftc\" (UID: \"68a85369-16d8-457b-a852-1bd45827378e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:51.096401 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.096386 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklpc\" (UniqueName: \"kubernetes.io/projected/d64a4e93-2ba3-4bf5-8f28-c62796e2d05c-kube-api-access-hklpc\") pod \"insights-operator-5c847dcfc4-x8khq\" (UID: \"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c\") " pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.113615 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.113590 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" Jul 07 17:54:51.130820 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.130795 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" Jul 07 17:54:51.171145 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.170889 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h8zpz"] Jul 07 17:54:51.173607 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:51.173577 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bcc290_f56a_4710_b664_c6234385c9a2.slice/crio-a51f8c1d7f0d14567764468bbf0870a65995adbff567c42bb9dbfbd710e8a334 WatchSource:0}: Error finding container a51f8c1d7f0d14567764468bbf0870a65995adbff567c42bb9dbfbd710e8a334: Status 404 returned error can't find the container with id a51f8c1d7f0d14567764468bbf0870a65995adbff567c42bb9dbfbd710e8a334 Jul 07 17:54:51.181280 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.181252 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnkzv\" (UniqueName: \"kubernetes.io/projected/ababb634-f89d-41e2-aafb-f1d865334371-kube-api-access-gnkzv\") pod \"network-check-source-989f8b8dc-tz7ng\" (UID: \"ababb634-f89d-41e2-aafb-f1d865334371\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng" Jul 07 17:54:51.197598 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.197541 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnkzv\" (UniqueName: \"kubernetes.io/projected/ababb634-f89d-41e2-aafb-f1d865334371-kube-api-access-gnkzv\") pod \"network-check-source-989f8b8dc-tz7ng\" (UID: \"ababb634-f89d-41e2-aafb-f1d865334371\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng" Jul 07 17:54:51.290357 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.290305 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc"] Jul 07 17:54:51.294109 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:51.294085 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a85369_16d8_457b_a852_1bd45827378e.slice/crio-a32c8ec5ee334359e68ff2448625c88bd89ca0f1c959df6f31b77ff13492c8ee WatchSource:0}: Error finding container a32c8ec5ee334359e68ff2448625c88bd89ca0f1c959df6f31b77ff13492c8ee: Status 404 returned error can't find the container with id a32c8ec5ee334359e68ff2448625c88bd89ca0f1c959df6f31b77ff13492c8ee Jul 07 17:54:51.299912 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.299892 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng" Jul 07 17:54:51.315004 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.314981 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-x8khq"] Jul 07 17:54:51.318316 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:51.318297 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64a4e93_2ba3_4bf5_8f28_c62796e2d05c.slice/crio-a70b2ca17eb1809777b2db9c9df03106797adb86831902a3404f2a0e9e7c8973 WatchSource:0}: Error finding container a70b2ca17eb1809777b2db9c9df03106797adb86831902a3404f2a0e9e7c8973: Status 404 returned error can't find the container with id a70b2ca17eb1809777b2db9c9df03106797adb86831902a3404f2a0e9e7c8973 Jul 07 17:54:51.382496 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.382461 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:51.382650 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.382590 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:51.382650 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:51.382607 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:51.382751 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:51.382695 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls podName:c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:52.382673256 +0000 UTC m=+37.635789827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xlgtt" (UID: "c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:51.382751 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:51.382741 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:51.382849 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:51.382808 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls podName:29ef67eb-8543-4716-8f68-fe90e8cbe171 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:52.382789144 +0000 UTC m=+37.635905719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls") pod "dns-default-xd72b" (UID: "29ef67eb-8543-4716-8f68-fe90e8cbe171") : secret "dns-default-metrics-tls" not found Jul 07 17:54:51.459926 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.459896 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng"] Jul 07 17:54:51.463478 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:51.463444 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podababb634_f89d_41e2_aafb_f1d865334371.slice/crio-9a022f9d97721aac2a81bed8be7d5eea77d4721577fc7f03c07d35dd981b3152 WatchSource:0}: Error finding container 9a022f9d97721aac2a81bed8be7d5eea77d4721577fc7f03c07d35dd981b3152: Status 404 returned error can't find the container with id 9a022f9d97721aac2a81bed8be7d5eea77d4721577fc7f03c07d35dd981b3152 Jul 07 17:54:51.557605 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.557565 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng" event={"ID":"ababb634-f89d-41e2-aafb-f1d865334371","Type":"ContainerStarted","Data":"9a022f9d97721aac2a81bed8be7d5eea77d4721577fc7f03c07d35dd981b3152"} Jul 07 17:54:51.558802 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.558774 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" event={"ID":"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c","Type":"ContainerStarted","Data":"a70b2ca17eb1809777b2db9c9df03106797adb86831902a3404f2a0e9e7c8973"} Jul 07 17:54:51.560388 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.560358 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h8zpz" event={"ID":"62bcc290-f56a-4710-b664-c6234385c9a2","Type":"ContainerStarted","Data":"a51f8c1d7f0d14567764468bbf0870a65995adbff567c42bb9dbfbd710e8a334"} Jul 07 17:54:51.561597 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.561573 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" event={"ID":"68a85369-16d8-457b-a852-1bd45827378e","Type":"ContainerStarted","Data":"a32c8ec5ee334359e68ff2448625c88bd89ca0f1c959df6f31b77ff13492c8ee"} Jul 07 17:54:51.584096 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:51.584035 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:51.584216 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:51.584166 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:54:51.584275 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:51.584242 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls podName:88ce489b-b618-41d1-acbe-6fc6423af618 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:52.584219923 +0000 UTC m=+37.837336481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2fkvf" (UID: "88ce489b-b618-41d1-acbe-6fc6423af618") : secret "samples-operator-tls" not found Jul 07 17:54:52.218156 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.218118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:54:52.218460 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.218433 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:54:52.218640 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.218618 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:54:52.223871 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.223845 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:54:52.224119 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.224102 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:54:52.225950 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.225925 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g4wsg\"" Jul 07 17:54:52.226190 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.226173 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n28xq\"" Jul 07 17:54:52.404101 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.402777 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:52.404101 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.402869 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:52.404101 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:52.403093 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:52.404101 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:52.403159 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls podName:c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:54.403139089 +0000 UTC m=+39.656255666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xlgtt" (UID: "c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:52.404101 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:52.403600 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:52.404101 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:52.403645 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls podName:29ef67eb-8543-4716-8f68-fe90e8cbe171 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:54.403629676 +0000 UTC m=+39.656746250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls") pod "dns-default-xd72b" (UID: "29ef67eb-8543-4716-8f68-fe90e8cbe171") : secret "dns-default-metrics-tls" not found Jul 07 17:54:52.605232 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:52.604583 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:52.605232 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:52.604758 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:54:52.605232 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:52.604828 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls podName:88ce489b-b618-41d1-acbe-6fc6423af618 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:54.60480718 +0000 UTC m=+39.857923742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2fkvf" (UID: "88ce489b-b618-41d1-acbe-6fc6423af618") : secret "samples-operator-tls" not found Jul 07 17:54:54.421203 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:54.421172 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:54.421660 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:54.421234 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:54.421660 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:54.421291 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:54.421660 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:54.421317 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:54.421660 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:54.421379 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls podName:c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:58.421362318 +0000 UTC m=+43.674478887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xlgtt" (UID: "c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:54.421660 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:54.421395 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls podName:29ef67eb-8543-4716-8f68-fe90e8cbe171 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:58.421386953 +0000 UTC m=+43.674503511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls") pod "dns-default-xd72b" (UID: "29ef67eb-8543-4716-8f68-fe90e8cbe171") : secret "dns-default-metrics-tls" not found Jul 07 17:54:54.622740 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:54.622706 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:54.622919 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:54.622858 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:54:54.622980 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:54.622929 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls podName:88ce489b-b618-41d1-acbe-6fc6423af618 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:58.622913494 +0000 UTC m=+43.876030052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2fkvf" (UID: "88ce489b-b618-41d1-acbe-6fc6423af618") : secret "samples-operator-tls" not found Jul 07 17:54:57.566816 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.566779 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6"] Jul 07 17:54:57.583904 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.583872 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6" Jul 07 17:54:57.587547 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.587517 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:57.587716 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.587598 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4lblj\"" Jul 07 17:54:57.587716 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.587655 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 17:54:57.587716 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.587701 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6"] Jul 07 17:54:57.747254 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.747213 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttlbs\" (UniqueName: \"kubernetes.io/projected/03b7fcb4-f8d5-43ae-81da-5da0aabb8141-kube-api-access-ttlbs\") pod \"migrator-d57d58cc9-84js6\" (UID: \"03b7fcb4-f8d5-43ae-81da-5da0aabb8141\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6" Jul 07 17:54:57.848307 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.848227 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttlbs\" (UniqueName: \"kubernetes.io/projected/03b7fcb4-f8d5-43ae-81da-5da0aabb8141-kube-api-access-ttlbs\") pod \"migrator-d57d58cc9-84js6\" (UID: \"03b7fcb4-f8d5-43ae-81da-5da0aabb8141\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6" Jul 07 17:54:57.858948 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.858917 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttlbs\" (UniqueName: \"kubernetes.io/projected/03b7fcb4-f8d5-43ae-81da-5da0aabb8141-kube-api-access-ttlbs\") pod \"migrator-d57d58cc9-84js6\" (UID: \"03b7fcb4-f8d5-43ae-81da-5da0aabb8141\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6" Jul 07 17:54:57.893806 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:57.893769 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6" Jul 07 17:54:58.454081 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:58.454042 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:54:58.454274 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:58.454102 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:54:58.454274 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:58.454209 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:58.454274 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:58.454209 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:58.454274 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:58.454270 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls podName:29ef67eb-8543-4716-8f68-fe90e8cbe171 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:06.454252022 +0000 UTC m=+51.707368580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls") pod "dns-default-xd72b" (UID: "29ef67eb-8543-4716-8f68-fe90e8cbe171") : secret "dns-default-metrics-tls" not found Jul 07 17:54:58.454469 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:58.454287 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls podName:c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:06.454278218 +0000 UTC m=+51.707394776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xlgtt" (UID: "c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:54:58.655725 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:58.655693 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:54:58.656089 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:58.655858 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:54:58.656089 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:54:58.655928 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls podName:88ce489b-b618-41d1-acbe-6fc6423af618 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:06.655910807 +0000 UTC m=+51.909027368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2fkvf" (UID: "88ce489b-b618-41d1-acbe-6fc6423af618") : secret "samples-operator-tls" not found Jul 07 17:54:59.118920 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.118864 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6"] Jul 07 17:54:59.123294 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:54:59.123239 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b7fcb4_f8d5_43ae_81da_5da0aabb8141.slice/crio-be6a05c888bea0638d3aa72b7895345ed574b1e49f434b4d5c410020b66fb4d0 WatchSource:0}: Error finding container be6a05c888bea0638d3aa72b7895345ed574b1e49f434b4d5c410020b66fb4d0: Status 404 returned error can't find the container with id be6a05c888bea0638d3aa72b7895345ed574b1e49f434b4d5c410020b66fb4d0 Jul 07 17:54:59.584727 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.584691 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" event={"ID":"68a85369-16d8-457b-a852-1bd45827378e","Type":"ContainerStarted","Data":"29adb029feb92ecf4e3f9870a83f389354f353827b8a901b5ac487493706ffdb"} Jul 07 17:54:59.586020 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.585988 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6" event={"ID":"03b7fcb4-f8d5-43ae-81da-5da0aabb8141","Type":"ContainerStarted","Data":"be6a05c888bea0638d3aa72b7895345ed574b1e49f434b4d5c410020b66fb4d0"} Jul 07 17:54:59.587717 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.587687 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng" event={"ID":"ababb634-f89d-41e2-aafb-f1d865334371","Type":"ContainerStarted","Data":"879e1463fe8dc0d27d82de624b9d4397cf632aad8b49b9091675b5981dd2126b"} Jul 07 17:54:59.589392 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.589361 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" event={"ID":"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c","Type":"ContainerStarted","Data":"b3b7de10533374c967221d4aa293cf33d1ede4e2b0b88cf0cedae00ff097a28a"} Jul 07 17:54:59.592589 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.592563 2557 generic.go:356] "Generic (PLEG): container finished" podID="05818555-acf1-4dff-9440-e639a27413b9" containerID="c8061021b34d8c5b7b1806cb8b86bbbac436e34b261cb3b1d085693b82919a16" exitCode=0 Jul 07 17:54:59.592694 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.592640 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8npv" event={"ID":"05818555-acf1-4dff-9440-e639a27413b9","Type":"ContainerDied","Data":"c8061021b34d8c5b7b1806cb8b86bbbac436e34b261cb3b1d085693b82919a16"} Jul 07 17:54:59.594919 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.594143 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h8zpz" event={"ID":"62bcc290-f56a-4710-b664-c6234385c9a2","Type":"ContainerStarted","Data":"e493948eea3baf47316e40117dbfc817d9fd1b5f06b0af79697785132cbd3692"} Jul 07 17:54:59.602897 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.602858 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" podStartSLOduration=1.928400668 podStartE2EDuration="9.602844927s" totalImagesPullingTime="7.674444259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:59.601727539 +0000 UTC m=+44.854844120" watchObservedRunningTime="2026-07-07 17:54:59.602844927 +0000 UTC m=+44.855961506" Jul 07 17:54:59.644948 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.644890 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" podStartSLOduration=1.995341638 podStartE2EDuration="9.644869907s" totalImagesPullingTime="7.649528269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:59.644253995 +0000 UTC m=+44.897370576" watchObservedRunningTime="2026-07-07 17:54:59.644869907 +0000 UTC m=+44.897986488" Jul 07 17:54:59.666385 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.666307 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tz7ng" podStartSLOduration=2.150668231 podStartE2EDuration="9.666287293s" totalImagesPullingTime="7.515619062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:59.66437756 +0000 UTC m=+44.917494167" watchObservedRunningTime="2026-07-07 17:54:59.666287293 +0000 UTC m=+44.919403874" Jul 07 17:54:59.689077 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:54:59.689029 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h8zpz" podStartSLOduration=1.892764255 podStartE2EDuration="9.689006281s" totalImagesPullingTime="7.796242026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:59.688390857 +0000 UTC m=+44.941507465" watchObservedRunningTime="2026-07-07 17:54:59.689006281 +0000 UTC m=+44.942122862" Jul 07 17:55:00.451662 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.451630 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pqdtv"] Jul 07 17:55:00.473556 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.473524 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pqdtv"] Jul 07 17:55:00.473709 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.473660 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.477613 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.477588 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 17:55:00.477756 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.477731 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5jtj4\"" Jul 07 17:55:00.477859 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.477783 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 17:55:00.571776 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.571741 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/005a0572-6d40-4446-8df8-3dd9f7bd72ca-crio-socket\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.571969 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.571801 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78rzr\" (UniqueName: \"kubernetes.io/projected/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-api-access-78rzr\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.571969 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.571882 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.572087 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.571967 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/005a0572-6d40-4446-8df8-3dd9f7bd72ca-data-volume\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.572087 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.571994 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.600465 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.600429 2557 generic.go:356] "Generic (PLEG): container finished" podID="05818555-acf1-4dff-9440-e639a27413b9" containerID="e5af6957f706f835e9b0ca9b3f345fdda49f6eb1d935d2f1c7047055fedf4fc7" exitCode=0 Jul 07 17:55:00.600609 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.600507 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8npv" event={"ID":"05818555-acf1-4dff-9440-e639a27413b9","Type":"ContainerDied","Data":"e5af6957f706f835e9b0ca9b3f345fdda49f6eb1d935d2f1c7047055fedf4fc7"} Jul 07 17:55:00.672559 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.672519 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.672628 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/005a0572-6d40-4446-8df8-3dd9f7bd72ca-data-volume\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:00.672787 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.672810 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:00.672873 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:01.172851486 +0000 UTC m=+46.425968058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:00.672905 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.672945 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/005a0572-6d40-4446-8df8-3dd9f7bd72ca-crio-socket\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.672949 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/005a0572-6d40-4446-8df8-3dd9f7bd72ca-data-volume\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:00.672965 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:01.172947988 +0000 UTC m=+46.426064569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : configmap "kube-rbac-proxy" not found Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.673020 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78rzr\" (UniqueName: \"kubernetes.io/projected/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-api-access-78rzr\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.673665 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.673128 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/005a0572-6d40-4446-8df8-3dd9f7bd72ca-crio-socket\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:00.682562 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:00.682541 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78rzr\" (UniqueName: \"kubernetes.io/projected/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-api-access-78rzr\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:01.177843 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:01.177806 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:01.178018 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:01.177929 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:01.178018 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:01.177991 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:02.177971667 +0000 UTC m=+47.431088226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : configmap "kube-rbac-proxy" not found Jul 07 17:55:01.178018 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:01.177934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:01.178182 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:01.178024 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:01.178182 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:01.178092 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:02.17807605 +0000 UTC m=+47.431192612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:01.607713 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:01.607670 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8npv" event={"ID":"05818555-acf1-4dff-9440-e639a27413b9","Type":"ContainerStarted","Data":"70e66aa04faf35c7ecd7f93afcc85d78dd49997b7f4b17950af5f5de9d4ca4fa"} Jul 07 17:55:01.609440 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:01.609414 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6" event={"ID":"03b7fcb4-f8d5-43ae-81da-5da0aabb8141","Type":"ContainerStarted","Data":"ed9500031176dcdb94c03aedf055092dfde6f3a6ee309a3186bd2e0e96f0232f"} Jul 07 17:55:01.609554 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:01.609458 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6" event={"ID":"03b7fcb4-f8d5-43ae-81da-5da0aabb8141","Type":"ContainerStarted","Data":"33f938378c0b002a2ac4c494f7020a2ef5c3f8610d1db5adf57ee5111bf9e902"} Jul 07 17:55:01.636043 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:01.635998 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h8npv" podStartSLOduration=8.629780271 podStartE2EDuration="46.635984947s" totalImagesPullingTime="38.006204676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:01.634692218 +0000 UTC m=+46.887808797" watchObservedRunningTime="2026-07-07 17:55:01.635984947 +0000 UTC m=+46.889101528" Jul 07 17:55:01.659744 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:01.659698 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-84js6" podStartSLOduration=2.8875767039999998 podStartE2EDuration="4.65968413s" totalImagesPullingTime="1.772107426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:01.658570105 +0000 UTC m=+46.911686685" watchObservedRunningTime="2026-07-07 17:55:01.65968413 +0000 UTC m=+46.912800710" Jul 07 17:55:02.184711 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:02.184673 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:02.185102 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:02.184729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:02.185102 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:02.184826 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:02.185102 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:02.184840 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:02.185102 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:02.184889 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:04.184874845 +0000 UTC m=+49.437991406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:02.185102 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:02.184917 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:04.184907848 +0000 UTC m=+49.438024406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : configmap "kube-rbac-proxy" not found Jul 07 17:55:03.027261 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:03.027237 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52tkx_1fffc956-ac04-4a88-a353-a9aa99e3a8a3/dns-node-resolver/0.log" Jul 07 17:55:04.199849 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:04.199815 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:04.200223 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:04.199868 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:04.200223 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:04.199955 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:04.200223 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:04.199967 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:04.200223 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:04.200005 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:08.199993215 +0000 UTC m=+53.453109778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : configmap "kube-rbac-proxy" not found Jul 07 17:55:04.200223 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:04.200025 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:08.200011833 +0000 UTC m=+53.453128390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:04.427318 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:04.427290 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rnnrw_bfcd6894-b47b-4ecd-9dc2-38235226086c/node-ca/0.log" Jul 07 17:55:05.226708 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:05.226677 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h8zpz_62bcc290-f56a-4710-b664-c6234385c9a2/serve-healthcheck-canary/0.log" Jul 07 17:55:05.427237 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:05.427207 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-84js6_03b7fcb4-f8d5-43ae-81da-5da0aabb8141/migrator/0.log" Jul 07 17:55:05.628421 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:05.628390 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-84js6_03b7fcb4-f8d5-43ae-81da-5da0aabb8141/graceful-termination/0.log" Jul 07 17:55:06.515461 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:06.515424 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:55:06.515898 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:06.515482 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:55:06.515898 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:06.515580 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:55:06.515898 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:06.515634 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 17:55:06.515898 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:06.515648 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls podName:29ef67eb-8543-4716-8f68-fe90e8cbe171 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:22.515631279 +0000 UTC m=+67.768747840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls") pod "dns-default-xd72b" (UID: "29ef67eb-8543-4716-8f68-fe90e8cbe171") : secret "dns-default-metrics-tls" not found Jul 07 17:55:06.515898 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:06.515686 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls podName:c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:22.515670292 +0000 UTC m=+67.768786865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xlgtt" (UID: "c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95") : secret "cluster-monitoring-operator-tls" not found Jul 07 17:55:06.717220 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:06.717189 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:55:06.717432 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:06.717374 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 17:55:06.717507 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:06.717439 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls podName:88ce489b-b618-41d1-acbe-6fc6423af618 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:22.717423192 +0000 UTC m=+67.970539750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2fkvf" (UID: "88ce489b-b618-41d1-acbe-6fc6423af618") : secret "samples-operator-tls" not found Jul 07 17:55:08.226675 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:08.226645 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:08.227141 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:08.226700 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:08.227141 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:08.226756 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:08.227141 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:08.226802 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:16.226789333 +0000 UTC m=+61.479905891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : configmap "kube-rbac-proxy" not found Jul 07 17:55:08.227141 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:08.226807 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:08.227141 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:08.226859 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:16.226845779 +0000 UTC m=+61.479962338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:16.294105 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:16.294071 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:16.294625 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:16.294130 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:16.294625 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:16.294226 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:16.294625 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:16.294305 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls podName:005a0572-6d40-4446-8df8-3dd9f7bd72ca nodeName:}" failed. No retries permitted until 2026-07-07 17:55:32.294288555 +0000 UTC m=+77.547405117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pqdtv" (UID: "005a0572-6d40-4446-8df8-3dd9f7bd72ca") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:16.294756 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:16.294722 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/005a0572-6d40-4446-8df8-3dd9f7bd72ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:16.549050 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:16.548978 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7v76" Jul 07 17:55:20.024745 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.024708 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:55:20.025120 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.024804 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:55:20.028235 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.028210 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:55:20.028369 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.028211 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:55:20.038058 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.038028 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d207b749-497b-4625-9510-3c0855f6b245-metrics-certs\") pod \"network-metrics-daemon-7zfpw\" (UID: \"d207b749-497b-4625-9510-3c0855f6b245\") " pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:55:20.038178 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.038120 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd8e48b-8053-42a2-b2fb-777e20a75ed1-original-pull-secret\") pod \"global-pull-secret-syncer-6s26g\" (UID: \"1cd8e48b-8053-42a2-b2fb-777e20a75ed1\") " pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:55:20.125385 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.125353 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:55:20.127887 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.127866 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsgbp\" (UniqueName: \"kubernetes.io/projected/d5bc64ad-338f-4963-ad60-aedac86a2a61-kube-api-access-vsgbp\") pod \"network-check-target-dq27h\" (UID: \"d5bc64ad-338f-4963-ad60-aedac86a2a61\") " pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:55:20.134667 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.134645 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6s26g" Jul 07 17:55:20.147396 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.147373 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n28xq\"" Jul 07 17:55:20.155502 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.155477 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:55:20.156008 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.155993 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g4wsg\"" Jul 07 17:55:20.164631 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.163548 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zfpw" Jul 07 17:55:20.288028 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.287959 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6s26g"] Jul 07 17:55:20.291516 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:55:20.291475 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd8e48b_8053_42a2_b2fb_777e20a75ed1.slice/crio-4c8f8c91ecb0b5e6fba44e87226607d27986933eb92fc3ade0ebc85f3500b42f WatchSource:0}: Error finding container 4c8f8c91ecb0b5e6fba44e87226607d27986933eb92fc3ade0ebc85f3500b42f: Status 404 returned error can't find the container with id 4c8f8c91ecb0b5e6fba44e87226607d27986933eb92fc3ade0ebc85f3500b42f Jul 07 17:55:20.311115 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.311088 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dq27h"] Jul 07 17:55:20.314543 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:55:20.314508 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bc64ad_338f_4963_ad60_aedac86a2a61.slice/crio-40eaf16e63538c205401c85d360eddef12a82bb4c50b7b66423d15b3f0a75fe1 WatchSource:0}: Error finding container 40eaf16e63538c205401c85d360eddef12a82bb4c50b7b66423d15b3f0a75fe1: Status 404 returned error can't find the container with id 40eaf16e63538c205401c85d360eddef12a82bb4c50b7b66423d15b3f0a75fe1 Jul 07 17:55:20.336905 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.336882 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7zfpw"] Jul 07 17:55:20.340249 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:55:20.340225 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd207b749_497b_4625_9510_3c0855f6b245.slice/crio-dfb089f1c1e68ed0c535ce112b6a6ae13eea9f0989a8d3a5cd9bc4207d7d6ad0 WatchSource:0}: Error finding container dfb089f1c1e68ed0c535ce112b6a6ae13eea9f0989a8d3a5cd9bc4207d7d6ad0: Status 404 returned error can't find the container with id dfb089f1c1e68ed0c535ce112b6a6ae13eea9f0989a8d3a5cd9bc4207d7d6ad0 Jul 07 17:55:20.669663 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.669629 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dq27h" event={"ID":"d5bc64ad-338f-4963-ad60-aedac86a2a61","Type":"ContainerStarted","Data":"f6dad82e3d2bb900dfc34419b94b0113cf40edaf086df14cb79727e57b10cda1"} Jul 07 17:55:20.669869 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.669686 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dq27h" event={"ID":"d5bc64ad-338f-4963-ad60-aedac86a2a61","Type":"ContainerStarted","Data":"40eaf16e63538c205401c85d360eddef12a82bb4c50b7b66423d15b3f0a75fe1"} Jul 07 17:55:20.669869 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.669751 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:55:20.670767 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.670738 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6s26g" event={"ID":"1cd8e48b-8053-42a2-b2fb-777e20a75ed1","Type":"ContainerStarted","Data":"4c8f8c91ecb0b5e6fba44e87226607d27986933eb92fc3ade0ebc85f3500b42f"} Jul 07 17:55:20.671780 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.671761 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7zfpw" event={"ID":"d207b749-497b-4625-9510-3c0855f6b245","Type":"ContainerStarted","Data":"dfb089f1c1e68ed0c535ce112b6a6ae13eea9f0989a8d3a5cd9bc4207d7d6ad0"} Jul 07 17:55:20.687857 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:20.687812 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dq27h" podStartSLOduration=65.687798938 podStartE2EDuration="1m5.687798938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:20.686941309 +0000 UTC m=+65.940057890" watchObservedRunningTime="2026-07-07 17:55:20.687798938 +0000 UTC m=+65.940915518" Jul 07 17:55:21.677066 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:21.677024 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7zfpw" event={"ID":"d207b749-497b-4625-9510-3c0855f6b245","Type":"ContainerStarted","Data":"c2295202e3b42734a43e97f828a41df720d9df9234a851bf40bfd0fdfe2b9f5c"} Jul 07 17:55:21.677530 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:21.677088 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7zfpw" event={"ID":"d207b749-497b-4625-9510-3c0855f6b245","Type":"ContainerStarted","Data":"8f4d35140c9e120b93508ea1db0d6b01a0d80a979933f08b17c94376fa47f067"} Jul 07 17:55:21.698110 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:21.698063 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7zfpw" podStartSLOduration=64.758310544 podStartE2EDuration="1m5.698047571s" totalImagesPullingTime="939.737027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:21.696684295 +0000 UTC m=+66.949800878" watchObservedRunningTime="2026-07-07 17:55:21.698047571 +0000 UTC m=+66.951164150" Jul 07 17:55:22.545693 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.545654 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:55:22.545886 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.545717 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:55:22.548778 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.548749 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ef67eb-8543-4716-8f68-fe90e8cbe171-metrics-tls\") pod \"dns-default-xd72b\" (UID: \"29ef67eb-8543-4716-8f68-fe90e8cbe171\") " pod="openshift-dns/dns-default-xd72b" Jul 07 17:55:22.548878 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.548829 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xlgtt\" (UID: \"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:55:22.729696 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.729669 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dtpjm\"" Jul 07 17:55:22.736380 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.736347 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xd72b" Jul 07 17:55:22.747030 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.747008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:55:22.749886 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.749860 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88ce489b-b618-41d1-acbe-6fc6423af618-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2fkvf\" (UID: \"88ce489b-b618-41d1-acbe-6fc6423af618\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:55:22.787173 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.787049 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sqwtn\"" Jul 07 17:55:22.794829 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.794800 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" Jul 07 17:55:22.866182 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.866151 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf"] Jul 07 17:55:22.868590 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.868564 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:22.884993 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.884971 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 17:55:22.885843 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.885823 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 17:55:22.885944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.885864 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 17:55:22.885944 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.885863 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 17:55:22.886238 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.886220 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w"] Jul 07 17:55:22.888588 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.888564 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:22.905976 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:22.905938 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret" Jul 07 17:55:22.906249 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:22.906218 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-ca\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret" Jul 07 17:55:22.906249 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:22.906218 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret" Jul 07 17:55:22.907076 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:22.907050 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-service-proxy-server-certificates\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret" Jul 07 17:55:22.938131 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.938095 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf"] Jul 07 17:55:22.942000 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.941954 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w"] Jul 07 17:55:22.948410 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.948378 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2eaa0a6-5f8f-435e-8059-b472b3765605-tmp\") pod \"klusterlet-addon-workmgr-779dbcbb86-gzcvf\" (UID: \"b2eaa0a6-5f8f-435e-8059-b472b3765605\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:22.948536 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.948423 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-hub\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:22.948536 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.948507 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkff7\" (UniqueName: \"kubernetes.io/projected/d62b8d97-46bc-4c66-99b8-f9a507c0df40-kube-api-access-dkff7\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:22.948662 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.948545 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:22.948662 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.948575 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b2eaa0a6-5f8f-435e-8059-b472b3765605-klusterlet-config\") pod \"klusterlet-addon-workmgr-779dbcbb86-gzcvf\" (UID: \"b2eaa0a6-5f8f-435e-8059-b472b3765605\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:22.948662 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.948589 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grq9h\" (UniqueName: \"kubernetes.io/projected/b2eaa0a6-5f8f-435e-8059-b472b3765605-kube-api-access-grq9h\") pod \"klusterlet-addon-workmgr-779dbcbb86-gzcvf\" (UID: \"b2eaa0a6-5f8f-435e-8059-b472b3765605\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:22.948662 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.948614 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62b8d97-46bc-4c66-99b8-f9a507c0df40-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:22.948662 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.948651 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-ca\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:22.948845 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.948700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:22.950293 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.950271 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gp7cm\"" Jul 07 17:55:22.955293 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:22.955272 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" Jul 07 17:55:23.049855 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.049746 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkff7\" (UniqueName: \"kubernetes.io/projected/d62b8d97-46bc-4c66-99b8-f9a507c0df40-kube-api-access-dkff7\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:23.049855 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.049803 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:23.049855 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.049829 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b2eaa0a6-5f8f-435e-8059-b472b3765605-klusterlet-config\") pod \"klusterlet-addon-workmgr-779dbcbb86-gzcvf\" (UID: \"b2eaa0a6-5f8f-435e-8059-b472b3765605\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:23.049855 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.049853 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grq9h\" (UniqueName: \"kubernetes.io/projected/b2eaa0a6-5f8f-435e-8059-b472b3765605-kube-api-access-grq9h\") pod \"klusterlet-addon-workmgr-779dbcbb86-gzcvf\" (UID: \"b2eaa0a6-5f8f-435e-8059-b472b3765605\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:23.050182 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.049880 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62b8d97-46bc-4c66-99b8-f9a507c0df40-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:23.050182 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.049912 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-ca\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:23.050182 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.049936 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:23.050182 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.049954 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2eaa0a6-5f8f-435e-8059-b472b3765605-tmp\") pod \"klusterlet-addon-workmgr-779dbcbb86-gzcvf\" (UID: \"b2eaa0a6-5f8f-435e-8059-b472b3765605\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:23.050182 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.049973 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-hub\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:23.050434 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.050412 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2eaa0a6-5f8f-435e-8059-b472b3765605-tmp\") pod \"klusterlet-addon-workmgr-779dbcbb86-gzcvf\" (UID: \"b2eaa0a6-5f8f-435e-8059-b472b3765605\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:23.050694 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.050674 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62b8d97-46bc-4c66-99b8-f9a507c0df40-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:23.053037 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.053012 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b2eaa0a6-5f8f-435e-8059-b472b3765605-klusterlet-config\") pod \"klusterlet-addon-workmgr-779dbcbb86-gzcvf\" (UID: \"b2eaa0a6-5f8f-435e-8059-b472b3765605\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:23.065888 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.065858 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkff7\" (UniqueName: \"kubernetes.io/projected/d62b8d97-46bc-4c66-99b8-f9a507c0df40-kube-api-access-dkff7\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:23.068862 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.068837 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grq9h\" (UniqueName: \"kubernetes.io/projected/b2eaa0a6-5f8f-435e-8059-b472b3765605-kube-api-access-grq9h\") pod \"klusterlet-addon-workmgr-779dbcbb86-gzcvf\" (UID: \"b2eaa0a6-5f8f-435e-8059-b472b3765605\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:23.179657 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.179618 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:23.726845 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.726812 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 17:55:23.732919 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.732895 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:23.806910 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.806874 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 17:55:23.813717 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:23.813689 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-hub\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:24.051226 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:24.050774 2557 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-ca: failed to sync secret cache: timed out waiting for the condition Jul 07 17:55:24.051226 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:24.050865 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-ca podName:d62b8d97-46bc-4c66-99b8-f9a507c0df40 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:24.550842447 +0000 UTC m=+69.803959031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ca" (UniqueName: "kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-ca") pod "cluster-proxy-proxy-agent-6c65868c86-6qt5w" (UID: "d62b8d97-46bc-4c66-99b8-f9a507c0df40") : failed to sync secret cache: timed out waiting for the condition Jul 07 17:55:24.051226 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:24.051141 2557 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-hub-kubeconfig: failed to sync secret cache: timed out waiting for the condition Jul 07 17:55:24.051226 ip-10-0-137-242 kubenswrapper[2557]: E0707 17:55:24.051197 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-hub-kubeconfig podName:d62b8d97-46bc-4c66-99b8-f9a507c0df40 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:24.551180883 +0000 UTC m=+69.804297442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "hub-kubeconfig" (UniqueName: "kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-hub-kubeconfig") pod "cluster-proxy-proxy-agent-6c65868c86-6qt5w" (UID: "d62b8d97-46bc-4c66-99b8-f9a507c0df40") : failed to sync secret cache: timed out waiting for the condition Jul 07 17:55:24.116076 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:55:24.116036 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2eaa0a6_5f8f_435e_8059_b472b3765605.slice/crio-29805ca1d83593624774cc680128bd378a63709481218f8a1a18e19ece68bf3b WatchSource:0}: Error finding container 29805ca1d83593624774cc680128bd378a63709481218f8a1a18e19ece68bf3b: Status 404 returned error can't find the container with id 29805ca1d83593624774cc680128bd378a63709481218f8a1a18e19ece68bf3b Jul 07 17:55:24.117632 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.117608 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf"] Jul 07 17:55:24.123834 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.123795 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xd72b"] Jul 07 17:55:24.127820 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:55:24.127795 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ef67eb_8543_4716_8f68_fe90e8cbe171.slice/crio-7df9845994ffad7cf7fdc11ec504f08c386aa480259cc204e8fd0fb59e09efb5 WatchSource:0}: Error finding container 7df9845994ffad7cf7fdc11ec504f08c386aa480259cc204e8fd0fb59e09efb5: Status 404 returned error can't find the container with id 7df9845994ffad7cf7fdc11ec504f08c386aa480259cc204e8fd0fb59e09efb5 Jul 07 17:55:24.131485 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.131238 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf"] Jul 07 17:55:24.219536 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.219506 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 17:55:24.343392 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.343370 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt"] Jul 07 17:55:24.344974 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:55:24.344951 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bb7e59_91dc_4401_9e8a_a1d2bdf92c95.slice/crio-fd9c99586ced740e726c96c28d2705a68f0327bf278cd822f78a15d587fc1905 WatchSource:0}: Error finding container fd9c99586ced740e726c96c28d2705a68f0327bf278cd822f78a15d587fc1905: Status 404 returned error can't find the container with id fd9c99586ced740e726c96c28d2705a68f0327bf278cd822f78a15d587fc1905 Jul 07 17:55:24.492568 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.492538 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 17:55:24.563438 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.563391 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-ca\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:24.563438 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.563436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:24.566277 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.566222 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-ca\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:24.566586 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.566559 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d62b8d97-46bc-4c66-99b8-f9a507c0df40-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c65868c86-6qt5w\" (UID: \"d62b8d97-46bc-4c66-99b8-f9a507c0df40\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:24.692192 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.692030 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" event={"ID":"88ce489b-b618-41d1-acbe-6fc6423af618","Type":"ContainerStarted","Data":"32e5aef48a50a1f2999b2a957ffe5f560ecb5b8b6edcf53aa5665a3910896ea1"} Jul 07 17:55:24.695360 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.694649 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" event={"ID":"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95","Type":"ContainerStarted","Data":"fd9c99586ced740e726c96c28d2705a68f0327bf278cd822f78a15d587fc1905"} Jul 07 17:55:24.698669 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.698597 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xd72b" event={"ID":"29ef67eb-8543-4716-8f68-fe90e8cbe171","Type":"ContainerStarted","Data":"7df9845994ffad7cf7fdc11ec504f08c386aa480259cc204e8fd0fb59e09efb5"} Jul 07 17:55:24.701287 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.700490 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6s26g" event={"ID":"1cd8e48b-8053-42a2-b2fb-777e20a75ed1","Type":"ContainerStarted","Data":"8320fe6421ed86b5a03ee44114d5a7d5622b4eac32e17c1f5be6db0c5d0fc101"} Jul 07 17:55:24.703404 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.703342 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" event={"ID":"b2eaa0a6-5f8f-435e-8059-b472b3765605","Type":"ContainerStarted","Data":"29805ca1d83593624774cc680128bd378a63709481218f8a1a18e19ece68bf3b"} Jul 07 17:55:24.706760 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.706470 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" Jul 07 17:55:24.732778 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.731562 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6s26g" podStartSLOduration=65.066320713 podStartE2EDuration="1m8.731538867s" totalImagesPullingTime="3.665218154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:24.730903746 +0000 UTC m=+69.984020332" watchObservedRunningTime="2026-07-07 17:55:24.731538867 +0000 UTC m=+69.984655440" Jul 07 17:55:24.916529 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:24.916421 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w"] Jul 07 17:55:25.709915 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:25.709856 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" event={"ID":"d62b8d97-46bc-4c66-99b8-f9a507c0df40","Type":"ContainerStarted","Data":"cb3cd4b985bcf252fef74cd7e645b9a409370713466f3880777262ad067ca2b3"} Jul 07 17:55:31.734872 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.734826 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" event={"ID":"d62b8d97-46bc-4c66-99b8-f9a507c0df40","Type":"ContainerStarted","Data":"32f0f47f0e2a4484126a2a33eb8914a76a825a14950e466e0b7ec01360774111"} Jul 07 17:55:31.735416 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.734882 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" event={"ID":"d62b8d97-46bc-4c66-99b8-f9a507c0df40","Type":"ContainerStarted","Data":"e0a6e95b4580c3d97143dd051cb8e23b7358bbecf6df68be9911aedac95cb8a3"} Jul 07 17:55:31.735416 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.734902 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" event={"ID":"d62b8d97-46bc-4c66-99b8-f9a507c0df40","Type":"ContainerStarted","Data":"05b4828915c5bdd280aec85312869edcb04d81adee635e04f73c32918d7fb1f1"} Jul 07 17:55:31.736371 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.736347 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" event={"ID":"b2eaa0a6-5f8f-435e-8059-b472b3765605","Type":"ContainerStarted","Data":"c79a2c25ab77e521f478544c0d65a482aed330f3df8ef7eb111e9e0843ce7db5"} Jul 07 17:55:31.738056 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.738034 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" event={"ID":"88ce489b-b618-41d1-acbe-6fc6423af618","Type":"ContainerStarted","Data":"0b082ef31f2216be746a14535fa6d225390851fadf7799877fc731bf8e85ec47"} Jul 07 17:55:31.738154 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.738082 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" event={"ID":"88ce489b-b618-41d1-acbe-6fc6423af618","Type":"ContainerStarted","Data":"43db4f676c675c9721601ce9aac21f002836faec46806440e496cf7be9692c5e"} Jul 07 17:55:31.739499 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.739479 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" event={"ID":"c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95","Type":"ContainerStarted","Data":"a58a21826a3c7315721f645fd4f5b32216d157c1f300c492bd8d9a680a1531ca"} Jul 07 17:55:31.741358 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.741340 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xd72b" event={"ID":"29ef67eb-8543-4716-8f68-fe90e8cbe171","Type":"ContainerStarted","Data":"166c1112119f63d8d1baecb49704e7b519dd88808e3edbb9ec9d642ff8fac23b"} Jul 07 17:55:31.741444 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.741373 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xd72b" event={"ID":"29ef67eb-8543-4716-8f68-fe90e8cbe171","Type":"ContainerStarted","Data":"10a5f1d1d3ae54b6418b8eafe3c248d430649fe4f245e60c3fcc7f44c57c31ef"} Jul 07 17:55:31.741480 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.741445 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xd72b" Jul 07 17:55:31.780891 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.780840 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c65868c86-6qt5w" podStartSLOduration=3.647642728 podStartE2EDuration="9.780825841s" totalImagesPullingTime="6.133183113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:31.779791864 +0000 UTC m=+77.032908445" watchObservedRunningTime="2026-07-07 17:55:31.780825841 +0000 UTC m=+77.033942420" Jul 07 17:55:31.837082 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.837035 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2fkvf" podStartSLOduration=35.473074472 podStartE2EDuration="41.83702038s" totalImagesPullingTime="6.363945908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:31.836345934 +0000 UTC m=+77.089462509" watchObservedRunningTime="2026-07-07 17:55:31.83702038 +0000 UTC m=+77.090136960" Jul 07 17:55:31.872550 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.872494 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xd72b" podStartSLOduration=34.977196899 podStartE2EDuration="41.872476756s" totalImagesPullingTime="6.895279857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:31.871993446 +0000 UTC m=+77.125110027" watchObservedRunningTime="2026-07-07 17:55:31.872476756 +0000 UTC m=+77.125593335" Jul 07 17:55:31.941807 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:31.941754 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xlgtt" podStartSLOduration=35.262530825 podStartE2EDuration="41.941737776s" totalImagesPullingTime="6.679206951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:31.94048798 +0000 UTC m=+77.193604582" watchObservedRunningTime="2026-07-07 17:55:31.941737776 +0000 UTC m=+77.194854355" Jul 07 17:55:32.333432 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:32.333396 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:32.335849 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:32.335830 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/005a0572-6d40-4446-8df8-3dd9f7bd72ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pqdtv\" (UID: \"005a0572-6d40-4446-8df8-3dd9f7bd72ca\") " pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:32.590958 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:32.590877 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5jtj4\"" Jul 07 17:55:32.597337 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:32.597301 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pqdtv" Jul 07 17:55:32.745453 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:32.745421 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:32.746537 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:32.746520 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" Jul 07 17:55:32.771370 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:32.771297 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779dbcbb86-gzcvf" podStartSLOduration=3.83745239 podStartE2EDuration="10.771283362s" totalImagesPullingTime="6.933830972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:32.00706551 +0000 UTC m=+77.260182110" watchObservedRunningTime="2026-07-07 17:55:32.771283362 +0000 UTC m=+78.024399942" Jul 07 17:55:32.771837 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:32.771821 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pqdtv"] Jul 07 17:55:32.774454 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:55:32.774431 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005a0572_6d40_4446_8df8_3dd9f7bd72ca.slice/crio-561910dd181dc80d92cb932fb7dc89d005af3acd7ad883b1fb6992acec634e06 WatchSource:0}: Error finding container 561910dd181dc80d92cb932fb7dc89d005af3acd7ad883b1fb6992acec634e06: Status 404 returned error can't find the container with id 561910dd181dc80d92cb932fb7dc89d005af3acd7ad883b1fb6992acec634e06 Jul 07 17:55:33.750410 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:33.750365 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pqdtv" event={"ID":"005a0572-6d40-4446-8df8-3dd9f7bd72ca","Type":"ContainerStarted","Data":"561910dd181dc80d92cb932fb7dc89d005af3acd7ad883b1fb6992acec634e06"} Jul 07 17:55:37.767903 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:37.767865 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pqdtv" event={"ID":"005a0572-6d40-4446-8df8-3dd9f7bd72ca","Type":"ContainerStarted","Data":"c65871d82a7b4a7e6b3f38ad60aaa86c0e2ce8f7c458d9a3bcf9b17600d856a9"} Jul 07 17:55:37.768260 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:37.767931 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pqdtv" event={"ID":"005a0572-6d40-4446-8df8-3dd9f7bd72ca","Type":"ContainerStarted","Data":"dae44023c88fb5f84a1a0198cb910008a06f90104f30793ee91ddb64acc18052"} Jul 07 17:55:40.785369 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:40.785316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pqdtv" event={"ID":"005a0572-6d40-4446-8df8-3dd9f7bd72ca","Type":"ContainerStarted","Data":"197f414fbaf4fa98ec4fcc5ee6c707d5cd36a87ab5db4c8c68438f2390224fdd"} Jul 07 17:55:41.652479 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.652407 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pqdtv" podStartSLOduration=34.523477669 podStartE2EDuration="41.652384075s" totalImagesPullingTime="7.128906406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:55:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:40.819452278 +0000 UTC m=+86.072568857" watchObservedRunningTime="2026-07-07 17:55:41.652384075 +0000 UTC m=+86.905500656" Jul 07 17:55:41.652853 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.652828 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qwk6l"] Jul 07 17:55:41.655014 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.654995 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.658691 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.658671 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 17:55:41.658803 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.658784 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 17:55:41.659579 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.659565 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qjnn2\"" Jul 07 17:55:41.659630 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.659581 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 17:55:41.660187 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.660173 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 17:55:41.747703 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.747674 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xd72b" Jul 07 17:55:41.810681 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.810654 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.810681 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.810692 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-textfile\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.811139 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.810723 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b91fa988-036f-40d9-b1bf-6d62b311d082-root\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.811139 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.810750 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-wtmp\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.811139 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.810791 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgvk8\" (UniqueName: \"kubernetes.io/projected/b91fa988-036f-40d9-b1bf-6d62b311d082-kube-api-access-vgvk8\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.811139 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.810874 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b91fa988-036f-40d9-b1bf-6d62b311d082-sys\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.811139 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.810925 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-tls\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.811139 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.810948 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-accelerators-collector-config\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.811139 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.810980 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b91fa988-036f-40d9-b1bf-6d62b311d082-metrics-client-ca\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911385 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911291 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911385 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911341 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-textfile\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911598 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911446 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b91fa988-036f-40d9-b1bf-6d62b311d082-root\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911598 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911481 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-wtmp\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911598 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911557 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b91fa988-036f-40d9-b1bf-6d62b311d082-root\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911751 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911633 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgvk8\" (UniqueName: \"kubernetes.io/projected/b91fa988-036f-40d9-b1bf-6d62b311d082-kube-api-access-vgvk8\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911751 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911641 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-wtmp\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911751 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911670 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b91fa988-036f-40d9-b1bf-6d62b311d082-sys\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911751 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911691 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-textfile\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911751 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911726 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b91fa988-036f-40d9-b1bf-6d62b311d082-sys\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911990 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911816 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-tls\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911990 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911859 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-accelerators-collector-config\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.911990 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.911893 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b91fa988-036f-40d9-b1bf-6d62b311d082-metrics-client-ca\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.912310 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.912287 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-accelerators-collector-config\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.912472 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.912400 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b91fa988-036f-40d9-b1bf-6d62b311d082-metrics-client-ca\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.913866 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.913850 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.914282 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.914260 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b91fa988-036f-40d9-b1bf-6d62b311d082-node-exporter-tls\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.925428 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.925405 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgvk8\" (UniqueName: \"kubernetes.io/projected/b91fa988-036f-40d9-b1bf-6d62b311d082-kube-api-access-vgvk8\") pod \"node-exporter-qwk6l\" (UID: \"b91fa988-036f-40d9-b1bf-6d62b311d082\") " pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.964500 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:41.964472 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qwk6l" Jul 07 17:55:41.976050 ip-10-0-137-242 kubenswrapper[2557]: W0707 17:55:41.976025 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91fa988_036f_40d9_b1bf_6d62b311d082.slice/crio-ce374a2d3d502c1c0a336aabc0bde06959d79a9b73518886bb0e9dd19bcc0be6 WatchSource:0}: Error finding container ce374a2d3d502c1c0a336aabc0bde06959d79a9b73518886bb0e9dd19bcc0be6: Status 404 returned error can't find the container with id ce374a2d3d502c1c0a336aabc0bde06959d79a9b73518886bb0e9dd19bcc0be6 Jul 07 17:55:42.794954 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:42.794918 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwk6l" event={"ID":"b91fa988-036f-40d9-b1bf-6d62b311d082","Type":"ContainerStarted","Data":"ce374a2d3d502c1c0a336aabc0bde06959d79a9b73518886bb0e9dd19bcc0be6"} Jul 07 17:55:43.800387 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:43.800355 2557 generic.go:356] "Generic (PLEG): container finished" podID="b91fa988-036f-40d9-b1bf-6d62b311d082" containerID="3240b1c280d54dad11f179c26a4b78c40b13bf92ecfcbfd9e4aaa30a2434a3f5" exitCode=0 Jul 07 17:55:43.800761 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:43.800417 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwk6l" event={"ID":"b91fa988-036f-40d9-b1bf-6d62b311d082","Type":"ContainerDied","Data":"3240b1c280d54dad11f179c26a4b78c40b13bf92ecfcbfd9e4aaa30a2434a3f5"} Jul 07 17:55:44.805694 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:44.805661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwk6l" event={"ID":"b91fa988-036f-40d9-b1bf-6d62b311d082","Type":"ContainerStarted","Data":"5097cf335306eebd4d2e5e6e7a9673bc59577df4da2cb5ca80a89450e0a0fac8"} Jul 07 17:55:44.806073 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:44.805737 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwk6l" event={"ID":"b91fa988-036f-40d9-b1bf-6d62b311d082","Type":"ContainerStarted","Data":"74fa65b5c83159e04fd7e2e09e160943f55dfb0a4760037c103f45d15ff62b3d"} Jul 07 17:55:44.833635 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:44.833586 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qwk6l" podStartSLOduration=2.755325898 podStartE2EDuration="3.833569123s" totalImagesPullingTime="1.005876135s" totalInitContainerRuntime="72.36709ms" isStatefulPod=true podCreationTimestamp="2026-07-07 17:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:44.831844236 +0000 UTC m=+90.084960837" watchObservedRunningTime="2026-07-07 17:55:44.833569123 +0000 UTC m=+90.086685702" Jul 07 17:55:51.679911 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:55:51.679865 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dq27h" Jul 07 17:56:24.949031 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:56:24.948937 2557 generic.go:356] "Generic (PLEG): container finished" podID="d64a4e93-2ba3-4bf5-8f28-c62796e2d05c" containerID="b3b7de10533374c967221d4aa293cf33d1ede4e2b0b88cf0cedae00ff097a28a" exitCode=0 Jul 07 17:56:24.949031 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:56:24.949018 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" event={"ID":"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c","Type":"ContainerDied","Data":"b3b7de10533374c967221d4aa293cf33d1ede4e2b0b88cf0cedae00ff097a28a"} Jul 07 17:56:24.949503 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:56:24.949421 2557 scope.go:117] "RemoveContainer" containerID="b3b7de10533374c967221d4aa293cf33d1ede4e2b0b88cf0cedae00ff097a28a" Jul 07 17:56:25.953705 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:56:25.953669 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-x8khq" event={"ID":"d64a4e93-2ba3-4bf5-8f28-c62796e2d05c","Type":"ContainerStarted","Data":"18bf3a71f846cfef42a4bfab71b8220ffe44b47d44af7faa4a69f1933b0344fc"} Jul 07 17:56:29.968344 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:56:29.968280 2557 generic.go:356] "Generic (PLEG): container finished" podID="68a85369-16d8-457b-a852-1bd45827378e" containerID="29adb029feb92ecf4e3f9870a83f389354f353827b8a901b5ac487493706ffdb" exitCode=0 Jul 07 17:56:29.968802 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:56:29.968367 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" event={"ID":"68a85369-16d8-457b-a852-1bd45827378e","Type":"ContainerDied","Data":"29adb029feb92ecf4e3f9870a83f389354f353827b8a901b5ac487493706ffdb"} Jul 07 17:56:29.968802 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:56:29.968680 2557 scope.go:117] "RemoveContainer" containerID="29adb029feb92ecf4e3f9870a83f389354f353827b8a901b5ac487493706ffdb" Jul 07 17:56:30.973672 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:56:30.973632 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-psftc" event={"ID":"68a85369-16d8-457b-a852-1bd45827378e","Type":"ContainerStarted","Data":"e7fd3d04869cea2caf2abba86c4df36a0254101e2fb4c061c3146009c8ecbc0b"} Jul 07 17:59:15.081084 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:59:15.081053 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 17:59:15.090602 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:59:15.090557 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 17:59:15.099062 ip-10-0-137-242 kubenswrapper[2557]: I0707 17:59:15.099028 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:00:19.347773 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.347689 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6946598-l688g"] Jul 07 18:00:19.349885 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.349862 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.353833 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.353807 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 18:00:19.354771 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.354750 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 18:00:19.354889 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.354807 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 18:00:19.354889 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.354815 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 18:00:19.355072 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.355039 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sgjqx\"" Jul 07 18:00:19.356630 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.356494 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 18:00:19.392867 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.392841 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6946598-l688g"] Jul 07 18:00:19.460844 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.460811 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7f7p\" (UniqueName: \"kubernetes.io/projected/0458d2b7-6c91-4a55-b012-4a1e265ca008-kube-api-access-z7f7p\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.460844 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.460847 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0458d2b7-6c91-4a55-b012-4a1e265ca008-metrics-cert\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.461036 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.460922 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0458d2b7-6c91-4a55-b012-4a1e265ca008-cert\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.461036 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.460968 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0458d2b7-6c91-4a55-b012-4a1e265ca008-manager-config\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.562288 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.562257 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0458d2b7-6c91-4a55-b012-4a1e265ca008-cert\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.562412 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.562305 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0458d2b7-6c91-4a55-b012-4a1e265ca008-manager-config\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.562412 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.562358 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7f7p\" (UniqueName: \"kubernetes.io/projected/0458d2b7-6c91-4a55-b012-4a1e265ca008-kube-api-access-z7f7p\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.562484 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.562466 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0458d2b7-6c91-4a55-b012-4a1e265ca008-metrics-cert\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.562991 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.562965 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0458d2b7-6c91-4a55-b012-4a1e265ca008-manager-config\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.565074 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.565045 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0458d2b7-6c91-4a55-b012-4a1e265ca008-cert\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.565184 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.565084 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0458d2b7-6c91-4a55-b012-4a1e265ca008-metrics-cert\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.582031 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.582002 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7f7p\" (UniqueName: \"kubernetes.io/projected/0458d2b7-6c91-4a55-b012-4a1e265ca008-kube-api-access-z7f7p\") pod \"lws-controller-manager-58f6946598-l688g\" (UID: \"0458d2b7-6c91-4a55-b012-4a1e265ca008\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.659035 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.658942 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:19.800291 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.800259 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6946598-l688g"] Jul 07 18:00:19.803007 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:00:19.802971 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0458d2b7_6c91_4a55_b012_4a1e265ca008.slice/crio-82a5f67c627fe09a39b024cc5b409bcc880aa1ba7e096d02ca06f37634679230 WatchSource:0}: Error finding container 82a5f67c627fe09a39b024cc5b409bcc880aa1ba7e096d02ca06f37634679230: Status 404 returned error can't find the container with id 82a5f67c627fe09a39b024cc5b409bcc880aa1ba7e096d02ca06f37634679230 Jul 07 18:00:19.804898 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:19.804883 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:00:20.714823 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.713643 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw"] Jul 07 18:00:20.716283 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.716255 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.720790 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.720763 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 18:00:20.720902 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.720789 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 18:00:20.721071 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.721042 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 18:00:20.721183 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.721150 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 18:00:20.723226 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.722919 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sp2sj\"" Jul 07 18:00:20.735399 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.735370 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw"] Jul 07 18:00:20.772880 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.772840 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" event={"ID":"0458d2b7-6c91-4a55-b012-4a1e265ca008","Type":"ContainerStarted","Data":"82a5f67c627fe09a39b024cc5b409bcc880aa1ba7e096d02ca06f37634679230"} Jul 07 18:00:20.874769 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.874712 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a29d1b58-e125-452d-9c5b-9ca24a9066a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.874769 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.874761 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a29d1b58-e125-452d-9c5b-9ca24a9066a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.874970 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.874846 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a29d1b58-e125-452d-9c5b-9ca24a9066a0-tmp\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.874970 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.874890 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2tx9\" (UniqueName: \"kubernetes.io/projected/a29d1b58-e125-452d-9c5b-9ca24a9066a0-kube-api-access-m2tx9\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.874970 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.874925 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a29d1b58-e125-452d-9c5b-9ca24a9066a0-manifests\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.975823 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.975739 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a29d1b58-e125-452d-9c5b-9ca24a9066a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.975823 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.975775 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a29d1b58-e125-452d-9c5b-9ca24a9066a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.975823 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.975811 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a29d1b58-e125-452d-9c5b-9ca24a9066a0-tmp\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.976112 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.975847 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2tx9\" (UniqueName: \"kubernetes.io/projected/a29d1b58-e125-452d-9c5b-9ca24a9066a0-kube-api-access-m2tx9\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.976112 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.975875 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a29d1b58-e125-452d-9c5b-9ca24a9066a0-manifests\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.976333 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.976301 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a29d1b58-e125-452d-9c5b-9ca24a9066a0-manifests\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.978771 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.978746 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a29d1b58-e125-452d-9c5b-9ca24a9066a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.978885 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.978843 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a29d1b58-e125-452d-9c5b-9ca24a9066a0-tmp\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.979037 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.979021 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a29d1b58-e125-452d-9c5b-9ca24a9066a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:20.999347 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:20.999304 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2tx9\" (UniqueName: \"kubernetes.io/projected/a29d1b58-e125-452d-9c5b-9ca24a9066a0-kube-api-access-m2tx9\") pod \"opendatahub-operator-controller-manager-76cd65fb98-8qqlw\" (UID: \"a29d1b58-e125-452d-9c5b-9ca24a9066a0\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:21.031871 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:21.031841 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:21.194050 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:21.193928 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw"] Jul 07 18:00:22.050054 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:00:22.050021 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29d1b58_e125_452d_9c5b_9ca24a9066a0.slice/crio-76df10a7a66edc7f1f1b85f6d00d1dc0ee1620a3945fc28b6243a7d8a3975802 WatchSource:0}: Error finding container 76df10a7a66edc7f1f1b85f6d00d1dc0ee1620a3945fc28b6243a7d8a3975802: Status 404 returned error can't find the container with id 76df10a7a66edc7f1f1b85f6d00d1dc0ee1620a3945fc28b6243a7d8a3975802 Jul 07 18:00:22.783683 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:22.783644 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" event={"ID":"0458d2b7-6c91-4a55-b012-4a1e265ca008","Type":"ContainerStarted","Data":"4ecb94015251da8dfc9dec31898836ccd79d116b7d42880457b388f09ef4d468"} Jul 07 18:00:22.784013 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:22.783985 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:22.785729 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:22.785699 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" event={"ID":"a29d1b58-e125-452d-9c5b-9ca24a9066a0","Type":"ContainerStarted","Data":"76df10a7a66edc7f1f1b85f6d00d1dc0ee1620a3945fc28b6243a7d8a3975802"} Jul 07 18:00:22.807113 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:22.807058 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" podStartSLOduration=1.528986062 podStartE2EDuration="3.80704073s" totalImagesPullingTime="2.278054668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:22.803732075 +0000 UTC m=+368.056848682" watchObservedRunningTime="2026-07-07 18:00:22.80704073 +0000 UTC m=+368.060157309" Jul 07 18:00:24.795744 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:24.795713 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" event={"ID":"a29d1b58-e125-452d-9c5b-9ca24a9066a0","Type":"ContainerStarted","Data":"409b7eda45d5df363c4c5134064d2ec2445fa9790383dba2cef890509c44342e"} Jul 07 18:00:26.805505 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:26.805473 2557 generic.go:356] "Generic (PLEG): container finished" podID="a29d1b58-e125-452d-9c5b-9ca24a9066a0" containerID="409b7eda45d5df363c4c5134064d2ec2445fa9790383dba2cef890509c44342e" exitCode=0 Jul 07 18:00:26.805926 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:26.805543 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" event={"ID":"a29d1b58-e125-452d-9c5b-9ca24a9066a0","Type":"ContainerDied","Data":"409b7eda45d5df363c4c5134064d2ec2445fa9790383dba2cef890509c44342e"} Jul 07 18:00:27.810927 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:27.810891 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" event={"ID":"a29d1b58-e125-452d-9c5b-9ca24a9066a0","Type":"ContainerStarted","Data":"ccc9a1f645665e1e5422b991f45c964bb97537e43682c862a6fcb2d16cd39c2b"} Jul 07 18:00:27.811308 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:27.811116 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:27.838301 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:27.838221 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" podStartSLOduration=3.768030117 podStartE2EDuration="7.838200986s" totalImagesPullingTime="2.974853263s" totalInitContainerRuntime="1.095317606s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:27.837626538 +0000 UTC m=+373.090743139" watchObservedRunningTime="2026-07-07 18:00:27.838200986 +0000 UTC m=+373.091317594" Jul 07 18:00:33.795014 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:33.794982 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58f6946598-l688g" Jul 07 18:00:38.817631 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:38.817600 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-8qqlw" Jul 07 18:00:42.612855 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.612820 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb"] Jul 07 18:00:42.685727 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.685685 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb"] Jul 07 18:00:42.685886 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.685799 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.723913 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.723887 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 18:00:42.723913 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.723887 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zbg46\"" Jul 07 18:00:42.724178 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.724165 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 18:00:42.729612 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.729592 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e93bae-c22c-4bde-a2bf-dacfc26d91a8-tls-certs\") pod \"kube-auth-proxy-74ffcdc9c9-945sb\" (UID: \"98e93bae-c22c-4bde-a2bf-dacfc26d91a8\") " pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.729734 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.729621 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e93bae-c22c-4bde-a2bf-dacfc26d91a8-tmp\") pod \"kube-auth-proxy-74ffcdc9c9-945sb\" (UID: \"98e93bae-c22c-4bde-a2bf-dacfc26d91a8\") " pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.729734 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.729721 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflhv\" (UniqueName: \"kubernetes.io/projected/98e93bae-c22c-4bde-a2bf-dacfc26d91a8-kube-api-access-zflhv\") pod \"kube-auth-proxy-74ffcdc9c9-945sb\" (UID: \"98e93bae-c22c-4bde-a2bf-dacfc26d91a8\") " pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.746173 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.746150 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 18:00:42.746300 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.746228 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 18:00:42.830405 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.830367 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e93bae-c22c-4bde-a2bf-dacfc26d91a8-tls-certs\") pod \"kube-auth-proxy-74ffcdc9c9-945sb\" (UID: \"98e93bae-c22c-4bde-a2bf-dacfc26d91a8\") " pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.830602 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.830420 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e93bae-c22c-4bde-a2bf-dacfc26d91a8-tmp\") pod \"kube-auth-proxy-74ffcdc9c9-945sb\" (UID: \"98e93bae-c22c-4bde-a2bf-dacfc26d91a8\") " pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.830602 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.830496 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflhv\" (UniqueName: \"kubernetes.io/projected/98e93bae-c22c-4bde-a2bf-dacfc26d91a8-kube-api-access-zflhv\") pod \"kube-auth-proxy-74ffcdc9c9-945sb\" (UID: \"98e93bae-c22c-4bde-a2bf-dacfc26d91a8\") " pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.832961 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.832933 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e93bae-c22c-4bde-a2bf-dacfc26d91a8-tmp\") pod \"kube-auth-proxy-74ffcdc9c9-945sb\" (UID: \"98e93bae-c22c-4bde-a2bf-dacfc26d91a8\") " pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.833073 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.833058 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e93bae-c22c-4bde-a2bf-dacfc26d91a8-tls-certs\") pod \"kube-auth-proxy-74ffcdc9c9-945sb\" (UID: \"98e93bae-c22c-4bde-a2bf-dacfc26d91a8\") " pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.880305 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.880234 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflhv\" (UniqueName: \"kubernetes.io/projected/98e93bae-c22c-4bde-a2bf-dacfc26d91a8-kube-api-access-zflhv\") pod \"kube-auth-proxy-74ffcdc9c9-945sb\" (UID: \"98e93bae-c22c-4bde-a2bf-dacfc26d91a8\") " pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:42.995717 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:42.995684 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" Jul 07 18:00:43.234946 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:00:43.234904 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e93bae_c22c_4bde_a2bf_dacfc26d91a8.slice/crio-76f04d23550b85f3f98911b8009f6c5ef07f970afb11807943d366d152f00548 WatchSource:0}: Error finding container 76f04d23550b85f3f98911b8009f6c5ef07f970afb11807943d366d152f00548: Status 404 returned error can't find the container with id 76f04d23550b85f3f98911b8009f6c5ef07f970afb11807943d366d152f00548 Jul 07 18:00:43.242485 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:43.242460 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb"] Jul 07 18:00:43.875660 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:43.875619 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" event={"ID":"98e93bae-c22c-4bde-a2bf-dacfc26d91a8","Type":"ContainerStarted","Data":"76f04d23550b85f3f98911b8009f6c5ef07f970afb11807943d366d152f00548"} Jul 07 18:00:46.896940 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:46.896906 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" event={"ID":"98e93bae-c22c-4bde-a2bf-dacfc26d91a8","Type":"ContainerStarted","Data":"33d579a10666a133a089ec8ed1cedbe3a2559ccd82ed3ca7a4f8469897744ae9"} Jul 07 18:00:46.959587 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:00:46.959536 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-74ffcdc9c9-945sb" podStartSLOduration=2.069069838 podStartE2EDuration="4.959518523s" totalImagesPullingTime="2.890448685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:46.957282166 +0000 UTC m=+392.210398756" watchObservedRunningTime="2026-07-07 18:00:46.959518523 +0000 UTC m=+392.212635103" Jul 07 18:02:36.084673 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.084637 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv"] Jul 07 18:02:36.087608 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.087591 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.091040 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.091014 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 18:02:36.092017 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.091997 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 18:02:36.092125 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.092036 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 18:02:36.092125 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.092059 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xxlhd\"" Jul 07 18:02:36.092125 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.092109 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 18:02:36.113239 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.113213 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv"] Jul 07 18:02:36.216383 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.216305 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/92e795c1-96eb-4080-91d7-5dfae785f503-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.216575 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.216451 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/92e795c1-96eb-4080-91d7-5dfae785f503-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.216575 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.216518 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmlx4\" (UniqueName: \"kubernetes.io/projected/92e795c1-96eb-4080-91d7-5dfae785f503-kube-api-access-tmlx4\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.317761 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.317725 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/92e795c1-96eb-4080-91d7-5dfae785f503-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.317915 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.317776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/92e795c1-96eb-4080-91d7-5dfae785f503-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.317915 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.317820 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmlx4\" (UniqueName: \"kubernetes.io/projected/92e795c1-96eb-4080-91d7-5dfae785f503-kube-api-access-tmlx4\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.317988 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:02:36.317933 2557 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 07 18:02:36.318021 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:02:36.318013 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e795c1-96eb-4080-91d7-5dfae785f503-plugin-serving-cert podName:92e795c1-96eb-4080-91d7-5dfae785f503 nodeName:}" failed. No retries permitted until 2026-07-07 18:02:36.817992405 +0000 UTC m=+502.071108963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/92e795c1-96eb-4080-91d7-5dfae785f503-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-hs4gv" (UID: "92e795c1-96eb-4080-91d7-5dfae785f503") : secret "plugin-serving-cert" not found Jul 07 18:02:36.318430 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.318408 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/92e795c1-96eb-4080-91d7-5dfae785f503-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.347849 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.347773 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmlx4\" (UniqueName: \"kubernetes.io/projected/92e795c1-96eb-4080-91d7-5dfae785f503-kube-api-access-tmlx4\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.822669 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.822632 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/92e795c1-96eb-4080-91d7-5dfae785f503-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.825225 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.825205 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/92e795c1-96eb-4080-91d7-5dfae785f503-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hs4gv\" (UID: \"92e795c1-96eb-4080-91d7-5dfae785f503\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:36.997022 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:36.996988 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" Jul 07 18:02:37.147489 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:37.147458 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv"] Jul 07 18:02:37.151086 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:02:37.151057 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92e795c1_96eb_4080_91d7_5dfae785f503.slice/crio-f2e55cbd48d5cda7e98f245249b845bf2f0b7a68f527e3353c38c90922d85ee9 WatchSource:0}: Error finding container f2e55cbd48d5cda7e98f245249b845bf2f0b7a68f527e3353c38c90922d85ee9: Status 404 returned error can't find the container with id f2e55cbd48d5cda7e98f245249b845bf2f0b7a68f527e3353c38c90922d85ee9 Jul 07 18:02:37.329260 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:37.329175 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" event={"ID":"92e795c1-96eb-4080-91d7-5dfae785f503","Type":"ContainerStarted","Data":"f2e55cbd48d5cda7e98f245249b845bf2f0b7a68f527e3353c38c90922d85ee9"} Jul 07 18:02:47.758766 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:47.758730 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9"] Jul 07 18:02:47.785681 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:47.785638 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:02:47.884685 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:02:47.884625 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-xjfgd\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xjfgd\"" type="*v1.Secret" Jul 07 18:02:47.920028 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:47.919986 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/920f5599-1da7-40ab-9f5e-f791fcbd68c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tdwj9\" (UID: \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:02:47.920208 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:47.920081 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz67f\" (UniqueName: \"kubernetes.io/projected/920f5599-1da7-40ab-9f5e-f791fcbd68c6-kube-api-access-bz67f\") pod \"kuadrant-operator-controller-manager-86878c769f-tdwj9\" (UID: \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:02:47.968557 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:47.968525 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9"] Jul 07 18:02:48.021054 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:48.020975 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz67f\" (UniqueName: \"kubernetes.io/projected/920f5599-1da7-40ab-9f5e-f791fcbd68c6-kube-api-access-bz67f\") pod \"kuadrant-operator-controller-manager-86878c769f-tdwj9\" (UID: \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:02:48.021213 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:48.021063 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/920f5599-1da7-40ab-9f5e-f791fcbd68c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tdwj9\" (UID: \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:02:48.021455 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:48.021426 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/920f5599-1da7-40ab-9f5e-f791fcbd68c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tdwj9\" (UID: \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:02:48.045458 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:48.045428 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz67f\" (UniqueName: \"kubernetes.io/projected/920f5599-1da7-40ab-9f5e-f791fcbd68c6-kube-api-access-bz67f\") pod \"kuadrant-operator-controller-manager-86878c769f-tdwj9\" (UID: \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:02:48.762097 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:48.762059 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xjfgd\"" Jul 07 18:02:48.768749 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:48.768721 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:02:59.385854 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:59.385825 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9"] Jul 07 18:02:59.389138 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:02:59.389110 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920f5599_1da7_40ab_9f5e_f791fcbd68c6.slice/crio-4b38208aac4d48e44ab4eb3986ef0e0f254fa56ab3b83e69e4ce0f7eeffadbf1 WatchSource:0}: Error finding container 4b38208aac4d48e44ab4eb3986ef0e0f254fa56ab3b83e69e4ce0f7eeffadbf1: Status 404 returned error can't find the container with id 4b38208aac4d48e44ab4eb3986ef0e0f254fa56ab3b83e69e4ce0f7eeffadbf1 Jul 07 18:02:59.433277 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:59.433239 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" event={"ID":"92e795c1-96eb-4080-91d7-5dfae785f503","Type":"ContainerStarted","Data":"cad75797c645d3f34adbc8beec2a5832e915e5bd5ef96700a0401bdd401c6d14"} Jul 07 18:02:59.434649 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:59.434627 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" event={"ID":"920f5599-1da7-40ab-9f5e-f791fcbd68c6","Type":"ContainerStarted","Data":"4b38208aac4d48e44ab4eb3986ef0e0f254fa56ab3b83e69e4ce0f7eeffadbf1"} Jul 07 18:02:59.461960 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:02:59.461911 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hs4gv" podStartSLOduration=1.302893891 podStartE2EDuration="23.461892965s" totalImagesPullingTime="22.158999074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:02:59.460719501 +0000 UTC m=+524.713836079" watchObservedRunningTime="2026-07-07 18:02:59.461892965 +0000 UTC m=+524.715009545" Jul 07 18:03:08.475767 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:08.475727 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" event={"ID":"920f5599-1da7-40ab-9f5e-f791fcbd68c6","Type":"ContainerStarted","Data":"79bb830a3ba5519aa1179ffe9fa337ad2c202a8aacb227db5a47f5c039267ae8"} Jul 07 18:03:08.476298 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:08.475863 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:03:08.505341 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:08.505274 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" podStartSLOduration=13.356660201 podStartE2EDuration="21.505258936s" totalImagesPullingTime="8.148598735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:03:08.504038033 +0000 UTC m=+533.757154613" watchObservedRunningTime="2026-07-07 18:03:08.505258936 +0000 UTC m=+533.758375556" Jul 07 18:03:19.482743 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:19.482710 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:03:44.409513 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:44.409470 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hrdk"] Jul 07 18:03:44.411984 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:44.411952 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9hrdk" Jul 07 18:03:44.414754 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:44.414733 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bdkx7\"" Jul 07 18:03:44.419041 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:44.418901 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hrdk"] Jul 07 18:03:44.497932 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:44.497899 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hldhn\" (UniqueName: \"kubernetes.io/projected/bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3-kube-api-access-hldhn\") pod \"authorino-75d556bd77-9hrdk\" (UID: \"bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3\") " pod="kuadrant-system/authorino-75d556bd77-9hrdk" Jul 07 18:03:44.598986 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:44.598952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hldhn\" (UniqueName: \"kubernetes.io/projected/bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3-kube-api-access-hldhn\") pod \"authorino-75d556bd77-9hrdk\" (UID: \"bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3\") " pod="kuadrant-system/authorino-75d556bd77-9hrdk" Jul 07 18:03:44.611518 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:44.611486 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hldhn\" (UniqueName: \"kubernetes.io/projected/bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3-kube-api-access-hldhn\") pod \"authorino-75d556bd77-9hrdk\" (UID: \"bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3\") " pod="kuadrant-system/authorino-75d556bd77-9hrdk" Jul 07 18:03:44.721761 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:44.721675 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9hrdk" Jul 07 18:03:44.852612 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:44.852584 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hrdk"] Jul 07 18:03:44.855199 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:03:44.855161 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd5ad180_2c5d_43da_bf16_0f9c7be5fdd3.slice/crio-c110fb31db5ac476cf8404a37e427824bdad8728107c10236d0b589f1b0db01c WatchSource:0}: Error finding container c110fb31db5ac476cf8404a37e427824bdad8728107c10236d0b589f1b0db01c: Status 404 returned error can't find the container with id c110fb31db5ac476cf8404a37e427824bdad8728107c10236d0b589f1b0db01c Jul 07 18:03:45.629970 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:45.629932 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9hrdk" event={"ID":"bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3","Type":"ContainerStarted","Data":"c110fb31db5ac476cf8404a37e427824bdad8728107c10236d0b589f1b0db01c"} Jul 07 18:03:48.644664 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:48.644623 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9hrdk" event={"ID":"bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3","Type":"ContainerStarted","Data":"9493e8018db01aaf04853b77af0f9022f6c01756a97720edbf5a82a65b60b507"} Jul 07 18:03:48.667241 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:03:48.667190 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-9hrdk" podStartSLOduration=1.588939761 podStartE2EDuration="4.667175134s" totalImagesPullingTime="3.078235373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:03:48.66564759 +0000 UTC m=+573.918764170" watchObservedRunningTime="2026-07-07 18:03:48.667175134 +0000 UTC m=+573.920291715" Jul 07 18:04:15.108932 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:15.108907 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:04:15.121648 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:15.121625 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:04:15.607247 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:15.607211 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hrdk"] Jul 07 18:04:15.607563 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:15.607446 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-9hrdk" podUID="bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3" containerName="authorino" containerID="cri-o://9493e8018db01aaf04853b77af0f9022f6c01756a97720edbf5a82a65b60b507" gracePeriod=30 Jul 07 18:04:15.757690 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:15.757653 2557 generic.go:356] "Generic (PLEG): container finished" podID="bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3" containerID="9493e8018db01aaf04853b77af0f9022f6c01756a97720edbf5a82a65b60b507" exitCode=0 Jul 07 18:04:15.757884 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:15.757725 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9hrdk" event={"ID":"bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3","Type":"ContainerDied","Data":"9493e8018db01aaf04853b77af0f9022f6c01756a97720edbf5a82a65b60b507"} Jul 07 18:04:15.858948 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:15.858877 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9hrdk" Jul 07 18:04:15.941055 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:15.941018 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hldhn\" (UniqueName: \"kubernetes.io/projected/bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3-kube-api-access-hldhn\") pod \"bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3\" (UID: \"bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3\") " Jul 07 18:04:15.943462 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:15.943439 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3-kube-api-access-hldhn" (OuterVolumeSpecName: "kube-api-access-hldhn") pod "bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3" (UID: "bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3"). InnerVolumeSpecName "kube-api-access-hldhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:04:16.042303 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:16.042269 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hldhn\" (UniqueName: \"kubernetes.io/projected/bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3-kube-api-access-hldhn\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:04:16.762637 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:16.762608 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9hrdk" Jul 07 18:04:16.763118 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:16.762598 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9hrdk" event={"ID":"bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3","Type":"ContainerDied","Data":"c110fb31db5ac476cf8404a37e427824bdad8728107c10236d0b589f1b0db01c"} Jul 07 18:04:16.763118 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:16.762751 2557 scope.go:117] "RemoveContainer" containerID="9493e8018db01aaf04853b77af0f9022f6c01756a97720edbf5a82a65b60b507" Jul 07 18:04:16.784389 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:16.784355 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hrdk"] Jul 07 18:04:16.788536 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:16.788513 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hrdk"] Jul 07 18:04:17.223013 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:04:17.222978 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3" path="/var/lib/kubelet/pods/bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3/volumes" Jul 07 18:05:42.983720 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.983672 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr"] Jul 07 18:05:42.984385 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.984100 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3" containerName="authorino" Jul 07 18:05:42.984385 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.984118 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3" containerName="authorino" Jul 07 18:05:42.984385 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.984204 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd5ad180-2c5d-43da-bf16-0f9c7be5fdd3" containerName="authorino" Jul 07 18:05:42.987284 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.987260 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:42.991232 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.991203 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 07 18:05:42.991377 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.991203 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-45zl4\"" Jul 07 18:05:42.991377 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.991214 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 18:05:42.991377 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.991263 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 18:05:42.999164 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:42.999143 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr"] Jul 07 18:05:43.031231 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.031194 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.031231 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.031237 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.031468 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.031256 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dzp\" (UniqueName: \"kubernetes.io/projected/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-kube-api-access-r5dzp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.031468 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.031311 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.031468 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.031428 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.031468 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.031465 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.132128 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.132090 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.132128 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.132127 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dzp\" (UniqueName: \"kubernetes.io/projected/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-kube-api-access-r5dzp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.132382 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.132149 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.132382 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.132213 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.132382 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.132245 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.132382 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.132312 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.132592 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.132571 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.132629 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.132598 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.132676 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.132654 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.134752 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.134722 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.135014 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.134991 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.142455 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.142431 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dzp\" (UniqueName: \"kubernetes.io/projected/39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd-kube-api-access-r5dzp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k7rhr\" (UID: \"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.297706 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.297603 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:43.440390 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.440362 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr"] Jul 07 18:05:43.443438 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:05:43.443403 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a17f11_a9c9_4ff6_8e8b_5b37f4c501cd.slice/crio-f505825ec6005c8a37cdc040cb89ad524ec0b3b6092e8d8928eeb743319181e8 WatchSource:0}: Error finding container f505825ec6005c8a37cdc040cb89ad524ec0b3b6092e8d8928eeb743319181e8: Status 404 returned error can't find the container with id f505825ec6005c8a37cdc040cb89ad524ec0b3b6092e8d8928eeb743319181e8 Jul 07 18:05:43.444769 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:43.444740 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:05:44.108516 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:44.108483 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" event={"ID":"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd","Type":"ContainerStarted","Data":"f505825ec6005c8a37cdc040cb89ad524ec0b3b6092e8d8928eeb743319181e8"} Jul 07 18:05:45.114032 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:45.113995 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" event={"ID":"39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd","Type":"ContainerStarted","Data":"82c0ae8ad334904feba448c37885ca19764c78b43c795f976e7f552aa6d5064e"} Jul 07 18:05:45.134675 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:45.134619 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" podStartSLOduration=2.103667097 podStartE2EDuration="3.13459781s" totalImagesPullingTime="1.030930713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:05:45.13415951 +0000 UTC m=+690.387276090" watchObservedRunningTime="2026-07-07 18:05:45.13459781 +0000 UTC m=+690.387714393" Jul 07 18:05:53.298105 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:53.298071 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:53.298621 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:53.298137 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:53.310129 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:53.310106 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:05:54.161723 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:05:54.161684 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k7rhr" Jul 07 18:06:00.406004 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.405968 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj"] Jul 07 18:06:00.409601 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.409581 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.424205 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.424187 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 18:06:00.485698 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.485663 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfxk6\" (UniqueName: \"kubernetes.io/projected/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-kube-api-access-hfxk6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.485817 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.485700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.485817 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.485770 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.485817 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.485800 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.485922 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.485838 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.485922 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.485876 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.524851 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.524825 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj"] Jul 07 18:06:00.586933 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.586900 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.587094 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.586945 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.587094 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.586968 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfxk6\" (UniqueName: \"kubernetes.io/projected/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-kube-api-access-hfxk6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.587094 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.586997 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.587094 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.587051 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.587094 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.587076 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.587412 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.587389 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.587472 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.587434 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.587523 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.587481 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.589496 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.589476 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.589636 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.589620 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.632830 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.632799 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfxk6\" (UniqueName: \"kubernetes.io/projected/7b74bbc0-31f4-4b69-9052-d53b60b3cc0a-kube-api-access-hfxk6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj\" (UID: \"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.720039 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.719960 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:00.867772 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:00.867735 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj"] Jul 07 18:06:00.872854 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:06:00.872827 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b74bbc0_31f4_4b69_9052_d53b60b3cc0a.slice/crio-bd77123228dce3fd9a7552d25d343a309905d1a076213b0975d237a260dc56b5 WatchSource:0}: Error finding container bd77123228dce3fd9a7552d25d343a309905d1a076213b0975d237a260dc56b5: Status 404 returned error can't find the container with id bd77123228dce3fd9a7552d25d343a309905d1a076213b0975d237a260dc56b5 Jul 07 18:06:01.180746 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:01.180715 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" event={"ID":"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a","Type":"ContainerStarted","Data":"d0353045387eed094bf494a4d6bc9c5dbe74b888462f469513a4f81fe926eb00"} Jul 07 18:06:01.180906 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:01.180772 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" event={"ID":"7b74bbc0-31f4-4b69-9052-d53b60b3cc0a","Type":"ContainerStarted","Data":"bd77123228dce3fd9a7552d25d343a309905d1a076213b0975d237a260dc56b5"} Jul 07 18:06:10.720702 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:10.720612 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:10.720702 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:10.720686 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:10.733084 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:10.733055 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:10.762976 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:10.762924 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" podStartSLOduration=10.532865303 podStartE2EDuration="10.762908791s" totalImagesPullingTime="230.043488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:06:01.221181946 +0000 UTC m=+706.474298548" watchObservedRunningTime="2026-07-07 18:06:10.762908791 +0000 UTC m=+716.016025371" Jul 07 18:06:11.233675 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:11.233647 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj" Jul 07 18:06:23.381392 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.381357 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59"] Jul 07 18:06:23.387836 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.387814 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.390483 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.390460 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 18:06:23.398437 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.398415 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59"] Jul 07 18:06:23.486978 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.486943 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a773d55-505e-410d-ad7f-59a2a4aa152d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.487128 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.486983 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlpc\" (UniqueName: \"kubernetes.io/projected/7a773d55-505e-410d-ad7f-59a2a4aa152d-kube-api-access-znlpc\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.487128 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.487090 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.487128 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.487118 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.487275 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.487140 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.487275 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.487220 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.588602 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.588566 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.588602 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.588602 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.588816 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.588630 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.588816 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.588667 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.588816 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.588721 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a773d55-505e-410d-ad7f-59a2a4aa152d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.588816 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.588758 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlpc\" (UniqueName: \"kubernetes.io/projected/7a773d55-505e-410d-ad7f-59a2a4aa152d-kube-api-access-znlpc\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.589034 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.589010 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.589090 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.589070 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.589132 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.589095 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.591015 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.590977 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a773d55-505e-410d-ad7f-59a2a4aa152d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.591251 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.591232 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a773d55-505e-410d-ad7f-59a2a4aa152d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.597219 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.597191 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlpc\" (UniqueName: \"kubernetes.io/projected/7a773d55-505e-410d-ad7f-59a2a4aa152d-kube-api-access-znlpc\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59\" (UID: \"7a773d55-505e-410d-ad7f-59a2a4aa152d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.697554 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.697465 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:23.842914 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:23.842734 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59"] Jul 07 18:06:23.845658 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:06:23.845627 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a773d55_505e_410d_ad7f_59a2a4aa152d.slice/crio-2f9269f83da602bac7e5b60122cc15a5be88d95be25fc6ed1d7714582a734e0d WatchSource:0}: Error finding container 2f9269f83da602bac7e5b60122cc15a5be88d95be25fc6ed1d7714582a734e0d: Status 404 returned error can't find the container with id 2f9269f83da602bac7e5b60122cc15a5be88d95be25fc6ed1d7714582a734e0d Jul 07 18:06:24.276043 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:24.276007 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" event={"ID":"7a773d55-505e-410d-ad7f-59a2a4aa152d","Type":"ContainerStarted","Data":"9b72fc2a7ef5f2c55864a5dd11151ff57b900008381dc08dc179db984c69f8ef"} Jul 07 18:06:24.276203 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:24.276053 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" event={"ID":"7a773d55-505e-410d-ad7f-59a2a4aa152d","Type":"ContainerStarted","Data":"2f9269f83da602bac7e5b60122cc15a5be88d95be25fc6ed1d7714582a734e0d"} Jul 07 18:06:24.295268 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:24.295212 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" podStartSLOduration=1.047508206 podStartE2EDuration="1.295190346s" totalImagesPullingTime="247.68214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:06:24.293590982 +0000 UTC m=+729.546707559" watchObservedRunningTime="2026-07-07 18:06:24.295190346 +0000 UTC m=+729.548306927" Jul 07 18:06:33.697732 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:33.697692 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:33.698099 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:33.697755 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:33.710362 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:33.710339 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:06:34.331743 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:06:34.331710 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59" Jul 07 18:09:15.135043 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:09:15.134962 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:09:15.145049 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:09:15.145027 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:13:19.055992 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:19.055957 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9"] Jul 07 18:13:19.056603 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:19.056189 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" podUID="920f5599-1da7-40ab-9f5e-f791fcbd68c6" containerName="manager" containerID="cri-o://79bb830a3ba5519aa1179ffe9fa337ad2c202a8aacb227db5a47f5c039267ae8" gracePeriod=10 Jul 07 18:13:22.038056 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:22.038025 2557 generic.go:356] "Generic (PLEG): container finished" podID="920f5599-1da7-40ab-9f5e-f791fcbd68c6" containerID="79bb830a3ba5519aa1179ffe9fa337ad2c202a8aacb227db5a47f5c039267ae8" exitCode=0 Jul 07 18:13:22.038387 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:22.038096 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" event={"ID":"920f5599-1da7-40ab-9f5e-f791fcbd68c6","Type":"ContainerDied","Data":"79bb830a3ba5519aa1179ffe9fa337ad2c202a8aacb227db5a47f5c039267ae8"} Jul 07 18:13:22.604623 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:22.604598 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:13:22.647739 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:22.647711 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/920f5599-1da7-40ab-9f5e-f791fcbd68c6-extensions-socket-volume\") pod \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\" (UID: \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\") " Jul 07 18:13:22.647882 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:22.647756 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz67f\" (UniqueName: \"kubernetes.io/projected/920f5599-1da7-40ab-9f5e-f791fcbd68c6-kube-api-access-bz67f\") pod \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\" (UID: \"920f5599-1da7-40ab-9f5e-f791fcbd68c6\") " Jul 07 18:13:22.648150 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:22.648128 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920f5599-1da7-40ab-9f5e-f791fcbd68c6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "920f5599-1da7-40ab-9f5e-f791fcbd68c6" (UID: "920f5599-1da7-40ab-9f5e-f791fcbd68c6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:13:22.650002 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:22.649981 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920f5599-1da7-40ab-9f5e-f791fcbd68c6-kube-api-access-bz67f" (OuterVolumeSpecName: "kube-api-access-bz67f") pod "920f5599-1da7-40ab-9f5e-f791fcbd68c6" (UID: "920f5599-1da7-40ab-9f5e-f791fcbd68c6"). InnerVolumeSpecName "kube-api-access-bz67f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:13:22.748650 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:22.748579 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/920f5599-1da7-40ab-9f5e-f791fcbd68c6-extensions-socket-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:13:22.748650 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:22.748602 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz67f\" (UniqueName: \"kubernetes.io/projected/920f5599-1da7-40ab-9f5e-f791fcbd68c6-kube-api-access-bz67f\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:13:23.043525 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:23.043445 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" Jul 07 18:13:23.043525 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:23.043460 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9" event={"ID":"920f5599-1da7-40ab-9f5e-f791fcbd68c6","Type":"ContainerDied","Data":"4b38208aac4d48e44ab4eb3986ef0e0f254fa56ab3b83e69e4ce0f7eeffadbf1"} Jul 07 18:13:23.043525 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:23.043526 2557 scope.go:117] "RemoveContainer" containerID="79bb830a3ba5519aa1179ffe9fa337ad2c202a8aacb227db5a47f5c039267ae8" Jul 07 18:13:23.064203 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:23.064179 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9"] Jul 07 18:13:23.068282 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:23.068262 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tdwj9"] Jul 07 18:13:23.223196 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:13:23.223164 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920f5599-1da7-40ab-9f5e-f791fcbd68c6" path="/var/lib/kubelet/pods/920f5599-1da7-40ab-9f5e-f791fcbd68c6/volumes" Jul 07 18:14:15.161937 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:15.161907 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:14:15.168057 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:15.168028 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:14:20.376862 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.376831 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6"] Jul 07 18:14:20.377234 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.377199 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920f5599-1da7-40ab-9f5e-f791fcbd68c6" containerName="manager" Jul 07 18:14:20.377234 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.377211 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="920f5599-1da7-40ab-9f5e-f791fcbd68c6" containerName="manager" Jul 07 18:14:20.377310 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.377264 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="920f5599-1da7-40ab-9f5e-f791fcbd68c6" containerName="manager" Jul 07 18:14:20.380554 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.380538 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:14:20.383616 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.383596 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xjfgd\"" Jul 07 18:14:20.396360 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.396312 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6"] Jul 07 18:14:20.420985 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.420960 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb22ceb1-efb9-448e-acde-2b954c0f96e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hzwz6\" (UID: \"bb22ceb1-efb9-448e-acde-2b954c0f96e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:14:20.421144 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.420999 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vqf\" (UniqueName: \"kubernetes.io/projected/bb22ceb1-efb9-448e-acde-2b954c0f96e7-kube-api-access-29vqf\") pod \"kuadrant-operator-controller-manager-86878c769f-hzwz6\" (UID: \"bb22ceb1-efb9-448e-acde-2b954c0f96e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:14:20.522393 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.522356 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb22ceb1-efb9-448e-acde-2b954c0f96e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hzwz6\" (UID: \"bb22ceb1-efb9-448e-acde-2b954c0f96e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:14:20.522571 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.522410 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vqf\" (UniqueName: \"kubernetes.io/projected/bb22ceb1-efb9-448e-acde-2b954c0f96e7-kube-api-access-29vqf\") pod \"kuadrant-operator-controller-manager-86878c769f-hzwz6\" (UID: \"bb22ceb1-efb9-448e-acde-2b954c0f96e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:14:20.522750 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.522727 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb22ceb1-efb9-448e-acde-2b954c0f96e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hzwz6\" (UID: \"bb22ceb1-efb9-448e-acde-2b954c0f96e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:14:20.531793 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.531765 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vqf\" (UniqueName: \"kubernetes.io/projected/bb22ceb1-efb9-448e-acde-2b954c0f96e7-kube-api-access-29vqf\") pod \"kuadrant-operator-controller-manager-86878c769f-hzwz6\" (UID: \"bb22ceb1-efb9-448e-acde-2b954c0f96e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:14:20.690575 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.690487 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:14:20.850151 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:20.850114 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6"] Jul 07 18:14:20.861839 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:14:20.861813 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb22ceb1_efb9_448e_acde_2b954c0f96e7.slice/crio-423ccfb68a81eb1b133a51a85276abaccf4ac47b931024820ddcdc269de4c4f8 WatchSource:0}: Error finding container 423ccfb68a81eb1b133a51a85276abaccf4ac47b931024820ddcdc269de4c4f8: Status 404 returned error can't find the container with id 423ccfb68a81eb1b133a51a85276abaccf4ac47b931024820ddcdc269de4c4f8 Jul 07 18:14:21.279857 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:21.279814 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" event={"ID":"bb22ceb1-efb9-448e-acde-2b954c0f96e7","Type":"ContainerStarted","Data":"1fd54cf9b77fecd683666a4ce446b73b6c551afb9a5c63a4406ac7826ec33777"} Jul 07 18:14:21.280081 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:21.279887 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" event={"ID":"bb22ceb1-efb9-448e-acde-2b954c0f96e7","Type":"ContainerStarted","Data":"423ccfb68a81eb1b133a51a85276abaccf4ac47b931024820ddcdc269de4c4f8"} Jul 07 18:14:21.280081 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:21.279918 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:14:21.310818 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:21.310763 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" podStartSLOduration=1.31074639 podStartE2EDuration="1.31074639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:14:21.309800108 +0000 UTC m=+1206.562916688" watchObservedRunningTime="2026-07-07 18:14:21.31074639 +0000 UTC m=+1206.563862969" Jul 07 18:14:32.286230 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:14:32.286195 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hzwz6" Jul 07 18:19:15.192162 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:19:15.192135 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:19:15.196215 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:19:15.196185 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:24:15.218471 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:24:15.218353 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:24:15.229592 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:24:15.218722 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:27:47.473310 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.473258 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4"] Jul 07 18:27:47.476870 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.476850 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.480408 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.480367 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-46d729\"/\"kube-root-ca.crt\"" Jul 07 18:27:47.480553 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.480436 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-46d729\"/\"test-model-46d729-kserve-self-signed-certs\"" Jul 07 18:27:47.480553 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.480367 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-46d729\"/\"openshift-service-ca.crt\"" Jul 07 18:27:47.480553 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.480382 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-46d729\"/\"default-dockercfg-lrrfm\"" Jul 07 18:27:47.485494 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.485477 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4"] Jul 07 18:27:47.514502 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.514473 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-tmp-dir\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.514643 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.514518 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-dshm\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.514643 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.514540 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-home\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.514643 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.514599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae70ae3c-7c12-4603-a366-687029eedc73-tls-certs\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.514643 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.514637 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-model-cache\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.514790 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.514655 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qftxn\" (UniqueName: \"kubernetes.io/projected/ae70ae3c-7c12-4603-a366-687029eedc73-kube-api-access-qftxn\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.615716 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.615673 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae70ae3c-7c12-4603-a366-687029eedc73-tls-certs\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.615716 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.615721 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-model-cache\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.615979 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.615747 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qftxn\" (UniqueName: \"kubernetes.io/projected/ae70ae3c-7c12-4603-a366-687029eedc73-kube-api-access-qftxn\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.615979 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.615783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-tmp-dir\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.615979 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.615823 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-dshm\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.615979 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.615849 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-home\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.616207 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.616186 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-model-cache\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.616263 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.616221 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-tmp-dir\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.616308 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.616278 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-home\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.618243 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.618219 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-dshm\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.618440 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.618424 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae70ae3c-7c12-4603-a366-687029eedc73-tls-certs\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.632434 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.632409 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qftxn\" (UniqueName: \"kubernetes.io/projected/ae70ae3c-7c12-4603-a366-687029eedc73-kube-api-access-qftxn\") pod \"test-model-46d729-kserve-5bc5f9c457-fhtc4\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.787914 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.787829 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:47.921052 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:47.921020 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4"] Jul 07 18:27:47.923098 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:27:47.923068 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae70ae3c_7c12_4603_a366_687029eedc73.slice/crio-83152a69cd1bbd399d83817f5e7f338dcabef9ac3a1df2d739206d461d1cdf1c WatchSource:0}: Error finding container 83152a69cd1bbd399d83817f5e7f338dcabef9ac3a1df2d739206d461d1cdf1c: Status 404 returned error can't find the container with id 83152a69cd1bbd399d83817f5e7f338dcabef9ac3a1df2d739206d461d1cdf1c Jul 07 18:27:48.655364 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:48.655312 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" event={"ID":"ae70ae3c-7c12-4603-a366-687029eedc73","Type":"ContainerStarted","Data":"43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440"} Jul 07 18:27:48.655731 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:48.655375 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" event={"ID":"ae70ae3c-7c12-4603-a366-687029eedc73","Type":"ContainerStarted","Data":"83152a69cd1bbd399d83817f5e7f338dcabef9ac3a1df2d739206d461d1cdf1c"} Jul 07 18:27:48.692115 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:48.692062 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" podStartSLOduration=1.69204495 podStartE2EDuration="1.69204495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:27:48.691634851 +0000 UTC m=+2013.944751432" watchObservedRunningTime="2026-07-07 18:27:48.69204495 +0000 UTC m=+2013.945161530" Jul 07 18:27:57.788415 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:57.788379 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:57.788415 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:57.788421 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:57.803517 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:57.803490 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:27:58.713473 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:27:58.713440 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:28:22.327038 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:22.326990 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4"] Jul 07 18:28:37.347530 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.347427 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" podUID="ae70ae3c-7c12-4603-a366-687029eedc73" containerName="main" containerID="cri-o://43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440" gracePeriod=45 Jul 07 18:28:37.597447 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.597422 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:28:37.751800 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.751708 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae70ae3c-7c12-4603-a366-687029eedc73-tls-certs\") pod \"ae70ae3c-7c12-4603-a366-687029eedc73\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " Jul 07 18:28:37.751800 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.751790 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qftxn\" (UniqueName: \"kubernetes.io/projected/ae70ae3c-7c12-4603-a366-687029eedc73-kube-api-access-qftxn\") pod \"ae70ae3c-7c12-4603-a366-687029eedc73\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " Jul 07 18:28:37.752013 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.751831 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-home\") pod \"ae70ae3c-7c12-4603-a366-687029eedc73\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " Jul 07 18:28:37.752013 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.751872 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-tmp-dir\") pod \"ae70ae3c-7c12-4603-a366-687029eedc73\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " Jul 07 18:28:37.752013 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.751901 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-dshm\") pod \"ae70ae3c-7c12-4603-a366-687029eedc73\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " Jul 07 18:28:37.752013 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.751959 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-model-cache\") pod \"ae70ae3c-7c12-4603-a366-687029eedc73\" (UID: \"ae70ae3c-7c12-4603-a366-687029eedc73\") " Jul 07 18:28:37.752221 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.752159 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-home" (OuterVolumeSpecName: "home") pod "ae70ae3c-7c12-4603-a366-687029eedc73" (UID: "ae70ae3c-7c12-4603-a366-687029eedc73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:28:37.752221 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.752197 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae70ae3c-7c12-4603-a366-687029eedc73" (UID: "ae70ae3c-7c12-4603-a366-687029eedc73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:28:37.752430 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.752404 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae70ae3c-7c12-4603-a366-687029eedc73" (UID: "ae70ae3c-7c12-4603-a366-687029eedc73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:28:37.754133 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.754104 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae70ae3c-7c12-4603-a366-687029eedc73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae70ae3c-7c12-4603-a366-687029eedc73" (UID: "ae70ae3c-7c12-4603-a366-687029eedc73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:28:37.754221 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.754173 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-dshm" (OuterVolumeSpecName: "dshm") pod "ae70ae3c-7c12-4603-a366-687029eedc73" (UID: "ae70ae3c-7c12-4603-a366-687029eedc73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:28:37.754221 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.754180 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae70ae3c-7c12-4603-a366-687029eedc73-kube-api-access-qftxn" (OuterVolumeSpecName: "kube-api-access-qftxn") pod "ae70ae3c-7c12-4603-a366-687029eedc73" (UID: "ae70ae3c-7c12-4603-a366-687029eedc73"). InnerVolumeSpecName "kube-api-access-qftxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:28:37.852615 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.852577 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-model-cache\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:28:37.852615 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.852608 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae70ae3c-7c12-4603-a366-687029eedc73-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:28:37.852615 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.852623 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qftxn\" (UniqueName: \"kubernetes.io/projected/ae70ae3c-7c12-4603-a366-687029eedc73-kube-api-access-qftxn\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:28:37.852884 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.852637 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-home\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:28:37.852884 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.852648 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-tmp-dir\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:28:37.852884 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.852660 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae70ae3c-7c12-4603-a366-687029eedc73-dshm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:28:37.872070 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.872029 2557 generic.go:356] "Generic (PLEG): container finished" podID="ae70ae3c-7c12-4603-a366-687029eedc73" containerID="43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440" exitCode=0 Jul 07 18:28:37.872209 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.872082 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" Jul 07 18:28:37.872209 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.872105 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" event={"ID":"ae70ae3c-7c12-4603-a366-687029eedc73","Type":"ContainerDied","Data":"43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440"} Jul 07 18:28:37.872209 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.872140 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4" event={"ID":"ae70ae3c-7c12-4603-a366-687029eedc73","Type":"ContainerDied","Data":"83152a69cd1bbd399d83817f5e7f338dcabef9ac3a1df2d739206d461d1cdf1c"} Jul 07 18:28:37.872209 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.872156 2557 scope.go:117] "RemoveContainer" containerID="43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440" Jul 07 18:28:37.881871 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.881846 2557 scope.go:117] "RemoveContainer" containerID="43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440" Jul 07 18:28:37.882147 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:28:37.882122 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440\": container with ID starting with 43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440 not found: ID does not exist" containerID="43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440" Jul 07 18:28:37.882210 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.882161 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440"} err="failed to get container status \"43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440\": rpc error: code = NotFound desc = could not find container \"43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440\": container with ID starting with 43d2739f1283aba1a364af44c0f928bb24d590dbddcc09c3f942fa3546683440 not found: ID does not exist" Jul 07 18:28:37.903253 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.903219 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4"] Jul 07 18:28:37.905928 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:37.905904 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-46d729/test-model-46d729-kserve-5bc5f9c457-fhtc4"] Jul 07 18:28:39.224223 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:28:39.224185 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae70ae3c-7c12-4603-a366-687029eedc73" path="/var/lib/kubelet/pods/ae70ae3c-7c12-4603-a366-687029eedc73/volumes" Jul 07 18:29:15.248526 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:29:15.248417 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:29:15.251757 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:29:15.248842 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:30:15.677783 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.677750 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd"] Jul 07 18:30:15.678293 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.678204 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae70ae3c-7c12-4603-a366-687029eedc73" containerName="main" Jul 07 18:30:15.678293 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.678222 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae70ae3c-7c12-4603-a366-687029eedc73" containerName="main" Jul 07 18:30:15.678293 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.678292 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae70ae3c-7c12-4603-a366-687029eedc73" containerName="main" Jul 07 18:30:15.681184 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.681160 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.683997 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.683962 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2b5160\"/\"auth-test-model-2b5160-kserve-self-signed-certs\"" Jul 07 18:30:15.685150 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.685128 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2b5160\"/\"kube-root-ca.crt\"" Jul 07 18:30:15.685419 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.685403 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2b5160\"/\"default-dockercfg-zn68h\"" Jul 07 18:30:15.685513 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.685405 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2b5160\"/\"openshift-service-ca.crt\"" Jul 07 18:30:15.691925 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.691902 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd"] Jul 07 18:30:15.730919 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.730888 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-home\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.731053 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.730924 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvjjb\" (UniqueName: \"kubernetes.io/projected/11cb892b-7eab-47ac-af13-91b255e0cdce-kube-api-access-gvjjb\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.731053 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.730943 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-tmp-dir\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.731053 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.731011 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-dshm\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.731053 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.731040 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-model-cache\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.731220 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.731075 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb892b-7eab-47ac-af13-91b255e0cdce-tls-certs\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.832056 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.832022 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-home\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.832056 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.832068 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjjb\" (UniqueName: \"kubernetes.io/projected/11cb892b-7eab-47ac-af13-91b255e0cdce-kube-api-access-gvjjb\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.832305 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.832088 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-tmp-dir\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.832305 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.832125 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-dshm\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.832305 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.832244 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-model-cache\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.832305 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.832295 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb892b-7eab-47ac-af13-91b255e0cdce-tls-certs\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.832597 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.832575 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-tmp-dir\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.832668 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.832597 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-home\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.832718 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.832697 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-model-cache\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.834608 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.834586 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-dshm\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.834920 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.834899 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb892b-7eab-47ac-af13-91b255e0cdce-tls-certs\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.841205 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.841179 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvjjb\" (UniqueName: \"kubernetes.io/projected/11cb892b-7eab-47ac-af13-91b255e0cdce-kube-api-access-gvjjb\") pod \"auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:15.992411 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:15.992301 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:16.128423 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:16.128287 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd"] Jul 07 18:30:16.130608 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:30:16.130584 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11cb892b_7eab_47ac_af13_91b255e0cdce.slice/crio-b9117fa4dd0ea153a93f12175ee2e7d135ec8c3ff81f09fc4890f110e4d105a1 WatchSource:0}: Error finding container b9117fa4dd0ea153a93f12175ee2e7d135ec8c3ff81f09fc4890f110e4d105a1: Status 404 returned error can't find the container with id b9117fa4dd0ea153a93f12175ee2e7d135ec8c3ff81f09fc4890f110e4d105a1 Jul 07 18:30:16.290816 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:16.290715 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" event={"ID":"11cb892b-7eab-47ac-af13-91b255e0cdce","Type":"ContainerStarted","Data":"44b17360407050471c999dc4edefb7b4f57497632822514adc9e6073d0356d41"} Jul 07 18:30:16.290816 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:16.290768 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" event={"ID":"11cb892b-7eab-47ac-af13-91b255e0cdce","Type":"ContainerStarted","Data":"b9117fa4dd0ea153a93f12175ee2e7d135ec8c3ff81f09fc4890f110e4d105a1"} Jul 07 18:30:16.312771 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:16.312706 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" podStartSLOduration=1.3126859739999999 podStartE2EDuration="1.312685974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:30:16.311081483 +0000 UTC m=+2161.564198063" watchObservedRunningTime="2026-07-07 18:30:16.312685974 +0000 UTC m=+2161.565802555" Jul 07 18:30:25.993341 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:25.993277 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:25.993820 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:25.993361 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:26.006031 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:26.006005 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:30:26.345474 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:30:26.345443 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:32:23.677710 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:23.677670 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd"] Jul 07 18:32:36.974908 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:36.974871 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt"] Jul 07 18:32:36.978084 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:36.978066 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:36.980829 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:36.980797 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2b5160\"/\"sub-test-model-2b5160-kserve-self-signed-certs\"" Jul 07 18:32:36.985903 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:36.985875 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt"] Jul 07 18:32:37.008125 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.008094 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tls-certs\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.008281 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.008154 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-dshm\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.008281 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.008197 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-model-cache\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.008281 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.008220 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5h6\" (UniqueName: \"kubernetes.io/projected/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-kube-api-access-sw5h6\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.008482 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.008347 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-home\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.008482 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.008393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tmp-dir\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.109114 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.109073 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-dshm\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.109306 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.109127 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-model-cache\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.109306 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.109146 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5h6\" (UniqueName: \"kubernetes.io/projected/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-kube-api-access-sw5h6\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.109306 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.109171 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-home\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.109306 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.109190 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tmp-dir\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.109306 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.109217 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tls-certs\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.109675 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.109638 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-home\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.109790 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.109652 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-model-cache\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.109790 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.109722 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tmp-dir\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.111605 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.111584 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-dshm\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.111808 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.111788 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tls-certs\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.117845 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.117814 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5h6\" (UniqueName: \"kubernetes.io/projected/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-kube-api-access-sw5h6\") pod \"sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.288369 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.288265 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:37.421402 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.421345 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt"] Jul 07 18:32:37.423900 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:32:37.423874 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde1e3881_1ef9_4605_9e21_fb67ff31e7b2.slice/crio-5ef3b8f56302a99c2d49de97dcf8c32bc1a68123d91b2ea2714050a1bf25a7ed WatchSource:0}: Error finding container 5ef3b8f56302a99c2d49de97dcf8c32bc1a68123d91b2ea2714050a1bf25a7ed: Status 404 returned error can't find the container with id 5ef3b8f56302a99c2d49de97dcf8c32bc1a68123d91b2ea2714050a1bf25a7ed Jul 07 18:32:37.888424 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.888378 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" event={"ID":"de1e3881-1ef9-4605-9e21-fb67ff31e7b2","Type":"ContainerStarted","Data":"83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46"} Jul 07 18:32:37.888606 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.888435 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" event={"ID":"de1e3881-1ef9-4605-9e21-fb67ff31e7b2","Type":"ContainerStarted","Data":"5ef3b8f56302a99c2d49de97dcf8c32bc1a68123d91b2ea2714050a1bf25a7ed"} Jul 07 18:32:37.905916 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:37.905870 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" podStartSLOduration=1.905853588 podStartE2EDuration="1.905853588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:32:37.905618943 +0000 UTC m=+2303.158735520" watchObservedRunningTime="2026-07-07 18:32:37.905853588 +0000 UTC m=+2303.158970168" Jul 07 18:32:38.698532 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:38.698485 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" podUID="11cb892b-7eab-47ac-af13-91b255e0cdce" containerName="main" containerID="cri-o://44b17360407050471c999dc4edefb7b4f57497632822514adc9e6073d0356d41" gracePeriod=45 Jul 07 18:32:38.894788 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:38.894756 2557 generic.go:356] "Generic (PLEG): container finished" podID="11cb892b-7eab-47ac-af13-91b255e0cdce" containerID="44b17360407050471c999dc4edefb7b4f57497632822514adc9e6073d0356d41" exitCode=0 Jul 07 18:32:38.894989 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:38.894841 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" event={"ID":"11cb892b-7eab-47ac-af13-91b255e0cdce","Type":"ContainerDied","Data":"44b17360407050471c999dc4edefb7b4f57497632822514adc9e6073d0356d41"} Jul 07 18:32:38.946944 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:38.946914 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:32:39.022576 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.022480 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb892b-7eab-47ac-af13-91b255e0cdce-tls-certs\") pod \"11cb892b-7eab-47ac-af13-91b255e0cdce\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " Jul 07 18:32:39.022576 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.022519 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-model-cache\") pod \"11cb892b-7eab-47ac-af13-91b255e0cdce\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " Jul 07 18:32:39.022576 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.022565 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-dshm\") pod \"11cb892b-7eab-47ac-af13-91b255e0cdce\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " Jul 07 18:32:39.022855 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.022727 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-home\") pod \"11cb892b-7eab-47ac-af13-91b255e0cdce\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " Jul 07 18:32:39.022855 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.022782 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvjjb\" (UniqueName: \"kubernetes.io/projected/11cb892b-7eab-47ac-af13-91b255e0cdce-kube-api-access-gvjjb\") pod \"11cb892b-7eab-47ac-af13-91b255e0cdce\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " Jul 07 18:32:39.022855 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.022813 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-tmp-dir\") pod \"11cb892b-7eab-47ac-af13-91b255e0cdce\" (UID: \"11cb892b-7eab-47ac-af13-91b255e0cdce\") " Jul 07 18:32:39.022855 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.022831 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-model-cache" (OuterVolumeSpecName: "model-cache") pod "11cb892b-7eab-47ac-af13-91b255e0cdce" (UID: "11cb892b-7eab-47ac-af13-91b255e0cdce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:32:39.023059 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.022930 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-home" (OuterVolumeSpecName: "home") pod "11cb892b-7eab-47ac-af13-91b255e0cdce" (UID: "11cb892b-7eab-47ac-af13-91b255e0cdce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:32:39.023059 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.023052 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-model-cache\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:32:39.023137 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.023065 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-home\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:32:39.023137 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.023110 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11cb892b-7eab-47ac-af13-91b255e0cdce" (UID: "11cb892b-7eab-47ac-af13-91b255e0cdce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:32:39.025012 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.024982 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cb892b-7eab-47ac-af13-91b255e0cdce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11cb892b-7eab-47ac-af13-91b255e0cdce" (UID: "11cb892b-7eab-47ac-af13-91b255e0cdce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:32:39.025012 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.024999 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-dshm" (OuterVolumeSpecName: "dshm") pod "11cb892b-7eab-47ac-af13-91b255e0cdce" (UID: "11cb892b-7eab-47ac-af13-91b255e0cdce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:32:39.025012 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.024978 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cb892b-7eab-47ac-af13-91b255e0cdce-kube-api-access-gvjjb" (OuterVolumeSpecName: "kube-api-access-gvjjb") pod "11cb892b-7eab-47ac-af13-91b255e0cdce" (UID: "11cb892b-7eab-47ac-af13-91b255e0cdce"). InnerVolumeSpecName "kube-api-access-gvjjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:32:39.123773 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.123732 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvjjb\" (UniqueName: \"kubernetes.io/projected/11cb892b-7eab-47ac-af13-91b255e0cdce-kube-api-access-gvjjb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:32:39.123773 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.123768 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-tmp-dir\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:32:39.123966 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.123784 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb892b-7eab-47ac-af13-91b255e0cdce-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:32:39.123966 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.123794 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cb892b-7eab-47ac-af13-91b255e0cdce-dshm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:32:39.900681 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.900647 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" event={"ID":"11cb892b-7eab-47ac-af13-91b255e0cdce","Type":"ContainerDied","Data":"b9117fa4dd0ea153a93f12175ee2e7d135ec8c3ff81f09fc4890f110e4d105a1"} Jul 07 18:32:39.901142 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.900692 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd" Jul 07 18:32:39.901142 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.900704 2557 scope.go:117] "RemoveContainer" containerID="44b17360407050471c999dc4edefb7b4f57497632822514adc9e6073d0356d41" Jul 07 18:32:39.916663 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.916622 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd"] Jul 07 18:32:39.918472 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:39.918451 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/auth-test-model-2b5160-kserve-6dfdf4df75-tfzzd"] Jul 07 18:32:41.223853 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:41.223820 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11cb892b-7eab-47ac-af13-91b255e0cdce" path="/var/lib/kubelet/pods/11cb892b-7eab-47ac-af13-91b255e0cdce/volumes" Jul 07 18:32:47.288484 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:47.288452 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:47.289074 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:47.288511 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:47.306263 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:47.306233 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:47.947494 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:47.947466 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:32:53.742362 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.742316 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp"] Jul 07 18:32:53.742745 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.742633 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cb892b-7eab-47ac-af13-91b255e0cdce" containerName="main" Jul 07 18:32:53.742745 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.742644 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cb892b-7eab-47ac-af13-91b255e0cdce" containerName="main" Jul 07 18:32:53.742745 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.742709 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cb892b-7eab-47ac-af13-91b255e0cdce" containerName="main" Jul 07 18:32:53.745525 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.745506 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.750472 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:32:53.750438 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"sub-test-model-200b05-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-200b05\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"sub-test-model-200b05-kserve-self-signed-certs\"" type="*v1.Secret" Jul 07 18:32:53.756722 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:32:53.756689 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"sub-test-model-200b05-kserve-bf689577-m6hfp\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-200b05\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" podUID="234190a9-a960-4952-a189-70ba79c4c66e" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.756821 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:32:53.756765 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-5hzlc\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-200b05\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"default-dockercfg-5hzlc\"" type="*v1.Secret" Jul 07 18:32:53.756961 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:32:53.756939 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-200b05\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 07 18:32:53.757176 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:32:53.757160 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-242.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-200b05\": no relationship found between node 'ip-10-0-137-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 07 18:32:53.808686 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.808653 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp"] Jul 07 18:32:53.826236 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.826211 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-home\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.826404 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.826240 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-dshm\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.826404 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.826281 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-tmp-dir\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.826404 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.826298 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234190a9-a960-4952-a189-70ba79c4c66e-tls-certs\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.826404 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.826387 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-model-cache\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.826563 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.826437 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htjvv\" (UniqueName: \"kubernetes.io/projected/234190a9-a960-4952-a189-70ba79c4c66e-kube-api-access-htjvv\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.927640 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.927602 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-tmp-dir\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.927640 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.927638 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234190a9-a960-4952-a189-70ba79c4c66e-tls-certs\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.927871 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.927675 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-model-cache\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.927871 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.927702 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htjvv\" (UniqueName: \"kubernetes.io/projected/234190a9-a960-4952-a189-70ba79c4c66e-kube-api-access-htjvv\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.927871 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.927729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-home\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.927871 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.927744 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-dshm\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.928084 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.928055 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-tmp-dir\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.928163 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.928138 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-model-cache\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.928213 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.928166 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-home\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:53.929978 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:53.929960 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-dshm\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:54.682564 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:54.682522 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"default-dockercfg-5hzlc\"" Jul 07 18:32:54.882241 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:54.882208 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"kube-root-ca.crt\"" Jul 07 18:32:54.928515 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:32:54.928484 2557 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition Jul 07 18:32:54.928664 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:32:54.928568 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234190a9-a960-4952-a189-70ba79c4c66e-tls-certs podName:234190a9-a960-4952-a189-70ba79c4c66e nodeName:}" failed. No retries permitted until 2026-07-07 18:32:55.428552418 +0000 UTC m=+2320.681668975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/234190a9-a960-4952-a189-70ba79c4c66e-tls-certs") pod "sub-test-model-200b05-kserve-bf689577-m6hfp" (UID: "234190a9-a960-4952-a189-70ba79c4c66e") : failed to sync secret cache: timed out waiting for the condition Jul 07 18:32:55.175690 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:55.175658 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"openshift-service-ca.crt\"" Jul 07 18:32:55.182405 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:55.182381 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htjvv\" (UniqueName: \"kubernetes.io/projected/234190a9-a960-4952-a189-70ba79c4c66e-kube-api-access-htjvv\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:55.310055 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:55.310024 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"sub-test-model-200b05-kserve-self-signed-certs\"" Jul 07 18:32:55.442150 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:55.442069 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234190a9-a960-4952-a189-70ba79c4c66e-tls-certs\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:55.444652 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:55.444623 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234190a9-a960-4952-a189-70ba79c4c66e-tls-certs\") pod \"sub-test-model-200b05-kserve-bf689577-m6hfp\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:55.554137 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:55.554097 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:32:55.705672 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:55.705585 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp"] Jul 07 18:32:55.708553 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:32:55.708529 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod234190a9_a960_4952_a189_70ba79c4c66e.slice/crio-d97fa356bb0534ca19ef3beeb2a698a5e7d60f0d913eb5ff6aa14aa49f2935d2 WatchSource:0}: Error finding container d97fa356bb0534ca19ef3beeb2a698a5e7d60f0d913eb5ff6aa14aa49f2935d2: Status 404 returned error can't find the container with id d97fa356bb0534ca19ef3beeb2a698a5e7d60f0d913eb5ff6aa14aa49f2935d2 Jul 07 18:32:55.978306 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:55.978225 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" event={"ID":"234190a9-a960-4952-a189-70ba79c4c66e","Type":"ContainerStarted","Data":"158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68"} Jul 07 18:32:55.978306 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:55.978266 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" event={"ID":"234190a9-a960-4952-a189-70ba79c4c66e","Type":"ContainerStarted","Data":"d97fa356bb0534ca19ef3beeb2a698a5e7d60f0d913eb5ff6aa14aa49f2935d2"} Jul 07 18:32:56.017381 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:32:56.017312 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" podStartSLOduration=3.017297025 podStartE2EDuration="3.017297025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:32:56.014639458 +0000 UTC m=+2321.267756039" watchObservedRunningTime="2026-07-07 18:32:56.017297025 +0000 UTC m=+2321.270413604" Jul 07 18:33:05.554863 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:33:05.554778 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:33:05.554863 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:33:05.554838 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:33:05.567144 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:33:05.567119 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:33:06.032559 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:33:06.032532 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:33:59.581136 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:33:59.581099 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt"] Jul 07 18:34:00.487684 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:00.487652 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp"] Jul 07 18:34:14.601876 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.601813 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" podUID="de1e3881-1ef9-4605-9e21-fb67ff31e7b2" containerName="main" containerID="cri-o://83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46" gracePeriod=45 Jul 07 18:34:14.841961 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.841932 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:34:14.941647 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.941560 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw5h6\" (UniqueName: \"kubernetes.io/projected/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-kube-api-access-sw5h6\") pod \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " Jul 07 18:34:14.941647 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.941630 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tls-certs\") pod \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " Jul 07 18:34:14.941973 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.941678 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-model-cache\") pod \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " Jul 07 18:34:14.941973 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.941708 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-home\") pod \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " Jul 07 18:34:14.941973 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.941731 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-dshm\") pod \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " Jul 07 18:34:14.941973 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.941751 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tmp-dir\") pod \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\" (UID: \"de1e3881-1ef9-4605-9e21-fb67ff31e7b2\") " Jul 07 18:34:14.941973 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.941959 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "de1e3881-1ef9-4605-9e21-fb67ff31e7b2" (UID: "de1e3881-1ef9-4605-9e21-fb67ff31e7b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:34:14.942213 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.941975 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-home" (OuterVolumeSpecName: "home") pod "de1e3881-1ef9-4605-9e21-fb67ff31e7b2" (UID: "de1e3881-1ef9-4605-9e21-fb67ff31e7b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:34:14.942213 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.942087 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de1e3881-1ef9-4605-9e21-fb67ff31e7b2" (UID: "de1e3881-1ef9-4605-9e21-fb67ff31e7b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:34:14.943847 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.943820 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-dshm" (OuterVolumeSpecName: "dshm") pod "de1e3881-1ef9-4605-9e21-fb67ff31e7b2" (UID: "de1e3881-1ef9-4605-9e21-fb67ff31e7b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:34:14.943975 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.943934 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-kube-api-access-sw5h6" (OuterVolumeSpecName: "kube-api-access-sw5h6") pod "de1e3881-1ef9-4605-9e21-fb67ff31e7b2" (UID: "de1e3881-1ef9-4605-9e21-fb67ff31e7b2"). InnerVolumeSpecName "kube-api-access-sw5h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:34:14.943975 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:14.943947 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de1e3881-1ef9-4605-9e21-fb67ff31e7b2" (UID: "de1e3881-1ef9-4605-9e21-fb67ff31e7b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:34:15.043156 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.043126 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.043156 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.043150 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-model-cache\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.043156 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.043159 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-home\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.043156 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.043167 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-dshm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.043492 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.043175 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-tmp-dir\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.043492 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.043183 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5h6\" (UniqueName: \"kubernetes.io/projected/de1e3881-1ef9-4605-9e21-fb67ff31e7b2-kube-api-access-sw5h6\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.272945 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.272819 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:34:15.282652 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.274959 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7v76_4f277f07-7a2c-4881-b1ba-aefff89d1279/ovn-acl-logging/0.log" Jul 07 18:34:15.324254 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.324218 2557 generic.go:356] "Generic (PLEG): container finished" podID="de1e3881-1ef9-4605-9e21-fb67ff31e7b2" containerID="83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46" exitCode=0 Jul 07 18:34:15.324447 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.324269 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" Jul 07 18:34:15.324447 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.324301 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" event={"ID":"de1e3881-1ef9-4605-9e21-fb67ff31e7b2","Type":"ContainerDied","Data":"83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46"} Jul 07 18:34:15.324447 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.324369 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt" event={"ID":"de1e3881-1ef9-4605-9e21-fb67ff31e7b2","Type":"ContainerDied","Data":"5ef3b8f56302a99c2d49de97dcf8c32bc1a68123d91b2ea2714050a1bf25a7ed"} Jul 07 18:34:15.324447 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.324389 2557 scope.go:117] "RemoveContainer" containerID="83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46" Jul 07 18:34:15.332697 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.332679 2557 scope.go:117] "RemoveContainer" containerID="83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46" Jul 07 18:34:15.332947 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:34:15.332929 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46\": container with ID starting with 83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46 not found: ID does not exist" containerID="83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46" Jul 07 18:34:15.333011 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.332955 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46"} err="failed to get container status \"83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46\": rpc error: code = NotFound desc = could not find container \"83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46\": container with ID starting with 83e571748920c63b706d7ecd6a5f072a688281cd037bc1498f1992427a549d46 not found: ID does not exist" Jul 07 18:34:15.342434 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.342409 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt"] Jul 07 18:34:15.346799 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.346781 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/sub-test-model-2b5160-kserve-75c7b7f8cf-vhxxt"] Jul 07 18:34:15.507647 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.507532 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" podUID="234190a9-a960-4952-a189-70ba79c4c66e" containerName="main" containerID="cri-o://158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68" gracePeriod=45 Jul 07 18:34:15.758669 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.758604 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:34:15.790085 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.790041 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7"] Jul 07 18:34:15.790366 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.790351 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1e3881-1ef9-4605-9e21-fb67ff31e7b2" containerName="main" Jul 07 18:34:15.790366 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.790366 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1e3881-1ef9-4605-9e21-fb67ff31e7b2" containerName="main" Jul 07 18:34:15.790512 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.790387 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234190a9-a960-4952-a189-70ba79c4c66e" containerName="main" Jul 07 18:34:15.790512 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.790392 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="234190a9-a960-4952-a189-70ba79c4c66e" containerName="main" Jul 07 18:34:15.790512 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.790441 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1e3881-1ef9-4605-9e21-fb67ff31e7b2" containerName="main" Jul 07 18:34:15.790512 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.790448 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="234190a9-a960-4952-a189-70ba79c4c66e" containerName="main" Jul 07 18:34:15.793033 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.793009 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.798805 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.798784 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2b5160\"/\"openshift-service-ca.crt\"" Jul 07 18:34:15.798925 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.798783 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2b5160\"/\"default-dockercfg-zn68h\"" Jul 07 18:34:15.798925 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.798784 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2b5160\"/\"rate-test-model-2b5160-kserve-self-signed-certs\"" Jul 07 18:34:15.798925 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.798788 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2b5160\"/\"kube-root-ca.crt\"" Jul 07 18:34:15.807855 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.807831 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7"] Jul 07 18:34:15.851125 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851093 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-dshm\") pod \"234190a9-a960-4952-a189-70ba79c4c66e\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " Jul 07 18:34:15.851296 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851132 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-model-cache\") pod \"234190a9-a960-4952-a189-70ba79c4c66e\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " Jul 07 18:34:15.851296 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851164 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234190a9-a960-4952-a189-70ba79c4c66e-tls-certs\") pod \"234190a9-a960-4952-a189-70ba79c4c66e\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " Jul 07 18:34:15.851296 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851182 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htjvv\" (UniqueName: \"kubernetes.io/projected/234190a9-a960-4952-a189-70ba79c4c66e-kube-api-access-htjvv\") pod \"234190a9-a960-4952-a189-70ba79c4c66e\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " Jul 07 18:34:15.851296 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851238 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-home\") pod \"234190a9-a960-4952-a189-70ba79c4c66e\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " Jul 07 18:34:15.851296 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851284 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-tmp-dir\") pod \"234190a9-a960-4952-a189-70ba79c4c66e\" (UID: \"234190a9-a960-4952-a189-70ba79c4c66e\") " Jul 07 18:34:15.851601 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851446 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-model-cache" (OuterVolumeSpecName: "model-cache") pod "234190a9-a960-4952-a189-70ba79c4c66e" (UID: "234190a9-a960-4952-a189-70ba79c4c66e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:34:15.851601 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851452 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-home\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.851601 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851514 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-model-cache\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.851601 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851587 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tmp-dir\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.851732 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851585 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-home" (OuterVolumeSpecName: "home") pod "234190a9-a960-4952-a189-70ba79c4c66e" (UID: "234190a9-a960-4952-a189-70ba79c4c66e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:34:15.851732 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851603 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "234190a9-a960-4952-a189-70ba79c4c66e" (UID: "234190a9-a960-4952-a189-70ba79c4c66e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:34:15.851732 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851647 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-dshm\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.851732 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851697 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tls-certs\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.851732 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851720 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqvhc\" (UniqueName: \"kubernetes.io/projected/55ca1b46-0787-4937-8aea-7c8a4d52ad26-kube-api-access-bqvhc\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.851912 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851825 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-model-cache\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.851912 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851846 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-home\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.851912 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.851858 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-tmp-dir\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.853723 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.853689 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-dshm" (OuterVolumeSpecName: "dshm") pod "234190a9-a960-4952-a189-70ba79c4c66e" (UID: "234190a9-a960-4952-a189-70ba79c4c66e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:34:15.853806 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.853748 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234190a9-a960-4952-a189-70ba79c4c66e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "234190a9-a960-4952-a189-70ba79c4c66e" (UID: "234190a9-a960-4952-a189-70ba79c4c66e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:34:15.853806 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.853764 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234190a9-a960-4952-a189-70ba79c4c66e-kube-api-access-htjvv" (OuterVolumeSpecName: "kube-api-access-htjvv") pod "234190a9-a960-4952-a189-70ba79c4c66e" (UID: "234190a9-a960-4952-a189-70ba79c4c66e"). InnerVolumeSpecName "kube-api-access-htjvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:34:15.952368 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952297 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-dshm\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.952368 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952372 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tls-certs\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.952670 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952391 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqvhc\" (UniqueName: \"kubernetes.io/projected/55ca1b46-0787-4937-8aea-7c8a4d52ad26-kube-api-access-bqvhc\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.952670 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952448 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-home\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.952670 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952475 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-model-cache\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.952670 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952527 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tmp-dir\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.952670 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952563 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234190a9-a960-4952-a189-70ba79c4c66e-dshm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.952670 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952577 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234190a9-a960-4952-a189-70ba79c4c66e-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.952670 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952591 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htjvv\" (UniqueName: \"kubernetes.io/projected/234190a9-a960-4952-a189-70ba79c4c66e-kube-api-access-htjvv\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:34:15.952998 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952893 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-home\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.952998 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952936 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-model-cache\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.952998 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.952946 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tmp-dir\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.954814 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.954795 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-dshm\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.955076 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.955058 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tls-certs\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:15.961399 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:15.961381 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqvhc\" (UniqueName: \"kubernetes.io/projected/55ca1b46-0787-4937-8aea-7c8a4d52ad26-kube-api-access-bqvhc\") pod \"rate-test-model-2b5160-kserve-58c9997f6c-cwqh7\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:16.102028 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.101993 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:16.241358 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.241300 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7"] Jul 07 18:34:16.243832 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:34:16.243805 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ca1b46_0787_4937_8aea_7c8a4d52ad26.slice/crio-4068f6d7c32a0d78cd2439cd07fe9180d394686dd18efc136b840766210efd50 WatchSource:0}: Error finding container 4068f6d7c32a0d78cd2439cd07fe9180d394686dd18efc136b840766210efd50: Status 404 returned error can't find the container with id 4068f6d7c32a0d78cd2439cd07fe9180d394686dd18efc136b840766210efd50 Jul 07 18:34:16.337449 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.337405 2557 generic.go:356] "Generic (PLEG): container finished" podID="234190a9-a960-4952-a189-70ba79c4c66e" containerID="158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68" exitCode=0 Jul 07 18:34:16.337625 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.337471 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" Jul 07 18:34:16.337625 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.337477 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" event={"ID":"234190a9-a960-4952-a189-70ba79c4c66e","Type":"ContainerDied","Data":"158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68"} Jul 07 18:34:16.337625 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.337521 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp" event={"ID":"234190a9-a960-4952-a189-70ba79c4c66e","Type":"ContainerDied","Data":"d97fa356bb0534ca19ef3beeb2a698a5e7d60f0d913eb5ff6aa14aa49f2935d2"} Jul 07 18:34:16.337625 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.337543 2557 scope.go:117] "RemoveContainer" containerID="158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68" Jul 07 18:34:16.339203 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.339177 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" event={"ID":"55ca1b46-0787-4937-8aea-7c8a4d52ad26","Type":"ContainerStarted","Data":"688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914"} Jul 07 18:34:16.339345 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.339219 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" event={"ID":"55ca1b46-0787-4937-8aea-7c8a4d52ad26","Type":"ContainerStarted","Data":"4068f6d7c32a0d78cd2439cd07fe9180d394686dd18efc136b840766210efd50"} Jul 07 18:34:16.346537 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.346468 2557 scope.go:117] "RemoveContainer" containerID="158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68" Jul 07 18:34:16.346914 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:34:16.346884 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68\": container with ID starting with 158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68 not found: ID does not exist" containerID="158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68" Jul 07 18:34:16.347008 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.346925 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68"} err="failed to get container status \"158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68\": rpc error: code = NotFound desc = could not find container \"158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68\": container with ID starting with 158a975fa77bc265d3158902c38fa698044a1db9838a57b616e11365541abb68 not found: ID does not exist" Jul 07 18:34:16.365109 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.365019 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" podStartSLOduration=1.365000408 podStartE2EDuration="1.365000408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:34:16.363690857 +0000 UTC m=+2401.616807449" watchObservedRunningTime="2026-07-07 18:34:16.365000408 +0000 UTC m=+2401.618116990" Jul 07 18:34:16.382904 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.382875 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp"] Jul 07 18:34:16.388876 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:16.388846 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/sub-test-model-200b05-kserve-bf689577-m6hfp"] Jul 07 18:34:17.224007 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:17.223970 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234190a9-a960-4952-a189-70ba79c4c66e" path="/var/lib/kubelet/pods/234190a9-a960-4952-a189-70ba79c4c66e/volumes" Jul 07 18:34:17.224397 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:17.224383 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de1e3881-1ef9-4605-9e21-fb67ff31e7b2" path="/var/lib/kubelet/pods/de1e3881-1ef9-4605-9e21-fb67ff31e7b2/volumes" Jul 07 18:34:26.103146 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:26.103101 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:26.103653 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:26.103162 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:26.115739 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:26.115714 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:26.394740 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:26.394650 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:34:34.358157 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.358122 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659"] Jul 07 18:34:34.361636 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.361612 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.365506 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.365475 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"default-dockercfg-5hzlc\"" Jul 07 18:34:34.365506 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.365504 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"openshift-service-ca.crt\"" Jul 07 18:34:34.365705 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.365547 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"rate-test-model-200b05-kserve-self-signed-certs\"" Jul 07 18:34:34.365705 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.365638 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-200b05\"/\"kube-root-ca.crt\"" Jul 07 18:34:34.373778 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.373758 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659"] Jul 07 18:34:34.420872 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.420850 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tmp-dir\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.420980 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.420876 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tls-certs\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.420980 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.420921 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-model-cache\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.420980 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.420967 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-dshm\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.421080 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.421006 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m22vg\" (UniqueName: \"kubernetes.io/projected/52427eae-93f5-4131-8ebb-5bde5f7fe74e-kube-api-access-m22vg\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.421080 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.421063 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-home\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.521426 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.521403 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-model-cache\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.521558 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.521437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-dshm\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.521558 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.521465 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22vg\" (UniqueName: \"kubernetes.io/projected/52427eae-93f5-4131-8ebb-5bde5f7fe74e-kube-api-access-m22vg\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.521558 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.521490 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-home\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.521558 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.521521 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tmp-dir\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.521558 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.521543 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tls-certs\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.521925 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.521891 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-model-cache\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.521925 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.521905 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tmp-dir\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.522065 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.521959 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-home\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.523887 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.523863 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-dshm\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.524151 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.524132 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tls-certs\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.532345 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.532290 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22vg\" (UniqueName: \"kubernetes.io/projected/52427eae-93f5-4131-8ebb-5bde5f7fe74e-kube-api-access-m22vg\") pod \"rate-test-model-200b05-kserve-665b65ff95-kl659\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.671865 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.671767 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:34.806622 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:34.806595 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659"] Jul 07 18:34:34.808249 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:34:34.808222 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52427eae_93f5_4131_8ebb_5bde5f7fe74e.slice/crio-ddbd073b1d17c2e43407d7d64db73fc8e1ecd7fec759adc89ba5225e542749d0 WatchSource:0}: Error finding container ddbd073b1d17c2e43407d7d64db73fc8e1ecd7fec759adc89ba5225e542749d0: Status 404 returned error can't find the container with id ddbd073b1d17c2e43407d7d64db73fc8e1ecd7fec759adc89ba5225e542749d0 Jul 07 18:34:35.424127 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:35.424044 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" event={"ID":"52427eae-93f5-4131-8ebb-5bde5f7fe74e","Type":"ContainerStarted","Data":"bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa"} Jul 07 18:34:35.424127 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:35.424085 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" event={"ID":"52427eae-93f5-4131-8ebb-5bde5f7fe74e","Type":"ContainerStarted","Data":"ddbd073b1d17c2e43407d7d64db73fc8e1ecd7fec759adc89ba5225e542749d0"} Jul 07 18:34:35.443359 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:35.443293 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" podStartSLOduration=1.443276762 podStartE2EDuration="1.443276762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:34:35.441224272 +0000 UTC m=+2420.694340853" watchObservedRunningTime="2026-07-07 18:34:35.443276762 +0000 UTC m=+2420.696393342" Jul 07 18:34:44.672682 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:44.672647 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:44.673179 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:44.672706 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:44.688969 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:44.688940 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:34:45.480259 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:34:45.480230 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:35:27.266712 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:27.266679 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7"] Jul 07 18:35:28.171526 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:28.171493 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659"] Jul 07 18:35:42.287501 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.287454 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" podUID="55ca1b46-0787-4937-8aea-7c8a4d52ad26" containerName="main" containerID="cri-o://688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914" gracePeriod=45 Jul 07 18:35:42.540649 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.540589 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:35:42.712708 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.712677 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-home\") pod \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " Jul 07 18:35:42.712889 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.712771 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-model-cache\") pod \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " Jul 07 18:35:42.712889 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.712800 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tls-certs\") pod \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " Jul 07 18:35:42.712889 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.712831 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-dshm\") pod \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " Jul 07 18:35:42.712889 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.712857 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tmp-dir\") pod \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " Jul 07 18:35:42.712889 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.712882 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqvhc\" (UniqueName: \"kubernetes.io/projected/55ca1b46-0787-4937-8aea-7c8a4d52ad26-kube-api-access-bqvhc\") pod \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\" (UID: \"55ca1b46-0787-4937-8aea-7c8a4d52ad26\") " Jul 07 18:35:42.713219 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.712964 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-home" (OuterVolumeSpecName: "home") pod "55ca1b46-0787-4937-8aea-7c8a4d52ad26" (UID: "55ca1b46-0787-4937-8aea-7c8a4d52ad26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:35:42.713219 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.713065 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-model-cache" (OuterVolumeSpecName: "model-cache") pod "55ca1b46-0787-4937-8aea-7c8a4d52ad26" (UID: "55ca1b46-0787-4937-8aea-7c8a4d52ad26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:35:42.713219 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.713152 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55ca1b46-0787-4937-8aea-7c8a4d52ad26" (UID: "55ca1b46-0787-4937-8aea-7c8a4d52ad26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:35:42.713219 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.713176 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-model-cache\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:42.713219 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.713194 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-home\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:42.715095 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.715063 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55ca1b46-0787-4937-8aea-7c8a4d52ad26" (UID: "55ca1b46-0787-4937-8aea-7c8a4d52ad26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:35:42.715214 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.715145 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ca1b46-0787-4937-8aea-7c8a4d52ad26-kube-api-access-bqvhc" (OuterVolumeSpecName: "kube-api-access-bqvhc") pod "55ca1b46-0787-4937-8aea-7c8a4d52ad26" (UID: "55ca1b46-0787-4937-8aea-7c8a4d52ad26"). InnerVolumeSpecName "kube-api-access-bqvhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:35:42.715390 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.715371 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-dshm" (OuterVolumeSpecName: "dshm") pod "55ca1b46-0787-4937-8aea-7c8a4d52ad26" (UID: "55ca1b46-0787-4937-8aea-7c8a4d52ad26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:35:42.718440 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.718406 2557 generic.go:356] "Generic (PLEG): container finished" podID="55ca1b46-0787-4937-8aea-7c8a4d52ad26" containerID="688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914" exitCode=0 Jul 07 18:35:42.718550 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.718454 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" Jul 07 18:35:42.718550 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.718482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" event={"ID":"55ca1b46-0787-4937-8aea-7c8a4d52ad26","Type":"ContainerDied","Data":"688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914"} Jul 07 18:35:42.718550 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.718525 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7" event={"ID":"55ca1b46-0787-4937-8aea-7c8a4d52ad26","Type":"ContainerDied","Data":"4068f6d7c32a0d78cd2439cd07fe9180d394686dd18efc136b840766210efd50"} Jul 07 18:35:42.718550 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.718546 2557 scope.go:117] "RemoveContainer" containerID="688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914" Jul 07 18:35:42.732489 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.732472 2557 scope.go:117] "RemoveContainer" containerID="688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914" Jul 07 18:35:42.732724 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:35:42.732707 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914\": container with ID starting with 688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914 not found: ID does not exist" containerID="688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914" Jul 07 18:35:42.732778 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.732732 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914"} err="failed to get container status \"688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914\": rpc error: code = NotFound desc = could not find container \"688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914\": container with ID starting with 688a5a93b119225d3801c1a6b189c714cdb99c481cb244470c1ed60431b5d914 not found: ID does not exist" Jul 07 18:35:42.748133 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.748106 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7"] Jul 07 18:35:42.749644 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.749621 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2b5160/rate-test-model-2b5160-kserve-58c9997f6c-cwqh7"] Jul 07 18:35:42.814439 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.814411 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:42.814439 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.814438 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-dshm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:42.814600 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.814450 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ca1b46-0787-4937-8aea-7c8a4d52ad26-tmp-dir\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:42.814600 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:42.814462 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqvhc\" (UniqueName: \"kubernetes.io/projected/55ca1b46-0787-4937-8aea-7c8a4d52ad26-kube-api-access-bqvhc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:43.191654 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.191540 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" podUID="52427eae-93f5-4131-8ebb-5bde5f7fe74e" containerName="main" containerID="cri-o://bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa" gracePeriod=45 Jul 07 18:35:43.223373 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.223344 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ca1b46-0787-4937-8aea-7c8a4d52ad26" path="/var/lib/kubelet/pods/55ca1b46-0787-4937-8aea-7c8a4d52ad26/volumes" Jul 07 18:35:43.438075 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.438051 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:35:43.520554 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520460 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-model-cache\") pod \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " Jul 07 18:35:43.520554 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520527 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m22vg\" (UniqueName: \"kubernetes.io/projected/52427eae-93f5-4131-8ebb-5bde5f7fe74e-kube-api-access-m22vg\") pod \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " Jul 07 18:35:43.520554 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520550 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-dshm\") pod \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " Jul 07 18:35:43.520830 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520594 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-home\") pod \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " Jul 07 18:35:43.520830 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520619 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tmp-dir\") pod \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " Jul 07 18:35:43.520830 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520642 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tls-certs\") pod \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\" (UID: \"52427eae-93f5-4131-8ebb-5bde5f7fe74e\") " Jul 07 18:35:43.520830 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520748 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-model-cache" (OuterVolumeSpecName: "model-cache") pod "52427eae-93f5-4131-8ebb-5bde5f7fe74e" (UID: "52427eae-93f5-4131-8ebb-5bde5f7fe74e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:35:43.521029 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520873 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-model-cache\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:43.521029 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520890 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-home" (OuterVolumeSpecName: "home") pod "52427eae-93f5-4131-8ebb-5bde5f7fe74e" (UID: "52427eae-93f5-4131-8ebb-5bde5f7fe74e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:35:43.521029 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.520931 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52427eae-93f5-4131-8ebb-5bde5f7fe74e" (UID: "52427eae-93f5-4131-8ebb-5bde5f7fe74e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:35:43.522939 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.522912 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52427eae-93f5-4131-8ebb-5bde5f7fe74e" (UID: "52427eae-93f5-4131-8ebb-5bde5f7fe74e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:35:43.523051 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.522953 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-dshm" (OuterVolumeSpecName: "dshm") pod "52427eae-93f5-4131-8ebb-5bde5f7fe74e" (UID: "52427eae-93f5-4131-8ebb-5bde5f7fe74e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:35:43.523051 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.522981 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52427eae-93f5-4131-8ebb-5bde5f7fe74e-kube-api-access-m22vg" (OuterVolumeSpecName: "kube-api-access-m22vg") pod "52427eae-93f5-4131-8ebb-5bde5f7fe74e" (UID: "52427eae-93f5-4131-8ebb-5bde5f7fe74e"). InnerVolumeSpecName "kube-api-access-m22vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:35:43.622179 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.622135 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m22vg\" (UniqueName: \"kubernetes.io/projected/52427eae-93f5-4131-8ebb-5bde5f7fe74e-kube-api-access-m22vg\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:43.622179 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.622172 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-dshm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:43.622179 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.622182 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-home\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:43.622179 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.622190 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tmp-dir\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:43.622179 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.622199 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52427eae-93f5-4131-8ebb-5bde5f7fe74e-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Jul 07 18:35:43.723946 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.723912 2557 generic.go:356] "Generic (PLEG): container finished" podID="52427eae-93f5-4131-8ebb-5bde5f7fe74e" containerID="bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa" exitCode=0 Jul 07 18:35:43.724122 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.723958 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" Jul 07 18:35:43.724122 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.723998 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" event={"ID":"52427eae-93f5-4131-8ebb-5bde5f7fe74e","Type":"ContainerDied","Data":"bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa"} Jul 07 18:35:43.724122 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.724034 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659" event={"ID":"52427eae-93f5-4131-8ebb-5bde5f7fe74e","Type":"ContainerDied","Data":"ddbd073b1d17c2e43407d7d64db73fc8e1ecd7fec759adc89ba5225e542749d0"} Jul 07 18:35:43.724122 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.724054 2557 scope.go:117] "RemoveContainer" containerID="bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa" Jul 07 18:35:43.732971 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.732953 2557 scope.go:117] "RemoveContainer" containerID="bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa" Jul 07 18:35:43.733253 ip-10-0-137-242 kubenswrapper[2557]: E0707 18:35:43.733235 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa\": container with ID starting with bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa not found: ID does not exist" containerID="bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa" Jul 07 18:35:43.733309 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.733263 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa"} err="failed to get container status \"bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa\": rpc error: code = NotFound desc = could not find container \"bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa\": container with ID starting with bd34504dfd2608f4e70fa9a18b2c181e9183e10e35836c099a8b0d853d71bdfa not found: ID does not exist" Jul 07 18:35:43.758507 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.758479 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659"] Jul 07 18:35:43.765524 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:43.765500 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-200b05/rate-test-model-200b05-kserve-665b65ff95-kl659"] Jul 07 18:35:45.223684 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:35:45.223651 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52427eae-93f5-4131-8ebb-5bde5f7fe74e" path="/var/lib/kubelet/pods/52427eae-93f5-4131-8ebb-5bde5f7fe74e/volumes" Jul 07 18:36:31.071135 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:31.071107 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cd65fb98-8qqlw_a29d1b58-e125-452d-9c5b-9ca24a9066a0/copy-manifests/0.log" Jul 07 18:36:31.089881 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:31.089856 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cd65fb98-8qqlw_a29d1b58-e125-452d-9c5b-9ca24a9066a0/manager/0.log" Jul 07 18:36:33.353272 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:33.353245 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hs4gv_92e795c1-96eb-4080-91d7-5dfae785f503/kuadrant-console-plugin/0.log" Jul 07 18:36:33.663194 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:33.663118 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hzwz6_bb22ceb1-efb9-448e-acde-2b954c0f96e7/manager/0.log" Jul 07 18:36:34.698843 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:34.698813 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74ffcdc9c9-945sb_98e93bae-c22c-4bde-a2bf-dacfc26d91a8/kube-auth-proxy/0.log" Jul 07 18:36:36.103816 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:36.103784 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-k7rhr_39a17f11-a9c9-4ff6-8e8b-5b37f4c501cd/main/0.log" Jul 07 18:36:36.239866 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:36.239836 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6dqkj_7b74bbc0-31f4-4b69-9052-d53b60b3cc0a/main/0.log" Jul 07 18:36:36.518456 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:36.518363 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-46w59_7a773d55-505e-410d-ad7f-59a2a4aa152d/main/0.log" Jul 07 18:36:40.858526 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.858493 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t248s/must-gather-vdjgj"] Jul 07 18:36:40.858896 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.858785 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ca1b46-0787-4937-8aea-7c8a4d52ad26" containerName="main" Jul 07 18:36:40.858896 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.858795 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ca1b46-0787-4937-8aea-7c8a4d52ad26" containerName="main" Jul 07 18:36:40.858896 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.858815 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52427eae-93f5-4131-8ebb-5bde5f7fe74e" containerName="main" Jul 07 18:36:40.858896 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.858820 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="52427eae-93f5-4131-8ebb-5bde5f7fe74e" containerName="main" Jul 07 18:36:40.858896 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.858869 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ca1b46-0787-4937-8aea-7c8a4d52ad26" containerName="main" Jul 07 18:36:40.858896 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.858879 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="52427eae-93f5-4131-8ebb-5bde5f7fe74e" containerName="main" Jul 07 18:36:40.861812 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.861795 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t248s/must-gather-vdjgj" Jul 07 18:36:40.868509 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.868490 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t248s\"/\"kube-root-ca.crt\"" Jul 07 18:36:40.868628 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.868519 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t248s\"/\"default-dockercfg-ml2d6\"" Jul 07 18:36:40.868628 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.868490 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t248s\"/\"openshift-service-ca.crt\"" Jul 07 18:36:40.948509 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.948447 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t248s/must-gather-vdjgj"] Jul 07 18:36:40.990018 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.989986 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/df76d70d-a2a1-4c36-85ed-4c21687888b5-must-gather-output\") pod \"must-gather-vdjgj\" (UID: \"df76d70d-a2a1-4c36-85ed-4c21687888b5\") " pod="openshift-must-gather-t248s/must-gather-vdjgj" Jul 07 18:36:40.990184 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:40.990048 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbhm\" (UniqueName: \"kubernetes.io/projected/df76d70d-a2a1-4c36-85ed-4c21687888b5-kube-api-access-9fbhm\") pod \"must-gather-vdjgj\" (UID: \"df76d70d-a2a1-4c36-85ed-4c21687888b5\") " pod="openshift-must-gather-t248s/must-gather-vdjgj" Jul 07 18:36:41.091108 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:41.091073 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbhm\" (UniqueName: \"kubernetes.io/projected/df76d70d-a2a1-4c36-85ed-4c21687888b5-kube-api-access-9fbhm\") pod \"must-gather-vdjgj\" (UID: \"df76d70d-a2a1-4c36-85ed-4c21687888b5\") " pod="openshift-must-gather-t248s/must-gather-vdjgj" Jul 07 18:36:41.091295 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:41.091122 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/df76d70d-a2a1-4c36-85ed-4c21687888b5-must-gather-output\") pod \"must-gather-vdjgj\" (UID: \"df76d70d-a2a1-4c36-85ed-4c21687888b5\") " pod="openshift-must-gather-t248s/must-gather-vdjgj" Jul 07 18:36:41.091445 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:41.091427 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/df76d70d-a2a1-4c36-85ed-4c21687888b5-must-gather-output\") pod \"must-gather-vdjgj\" (UID: \"df76d70d-a2a1-4c36-85ed-4c21687888b5\") " pod="openshift-must-gather-t248s/must-gather-vdjgj" Jul 07 18:36:41.111497 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:41.111434 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbhm\" (UniqueName: \"kubernetes.io/projected/df76d70d-a2a1-4c36-85ed-4c21687888b5-kube-api-access-9fbhm\") pod \"must-gather-vdjgj\" (UID: \"df76d70d-a2a1-4c36-85ed-4c21687888b5\") " pod="openshift-must-gather-t248s/must-gather-vdjgj" Jul 07 18:36:41.170500 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:41.170472 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t248s/must-gather-vdjgj" Jul 07 18:36:41.328357 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:41.328299 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t248s/must-gather-vdjgj"] Jul 07 18:36:41.331920 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:36:41.331894 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf76d70d_a2a1_4c36_85ed_4c21687888b5.slice/crio-c4b9d200428f494a905019dd15e68e4f823c1aac234b7b00cf2439d4e968573c WatchSource:0}: Error finding container c4b9d200428f494a905019dd15e68e4f823c1aac234b7b00cf2439d4e968573c: Status 404 returned error can't find the container with id c4b9d200428f494a905019dd15e68e4f823c1aac234b7b00cf2439d4e968573c Jul 07 18:36:41.333731 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:41.333708 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:36:41.966918 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:41.966885 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t248s/must-gather-vdjgj" event={"ID":"df76d70d-a2a1-4c36-85ed-4c21687888b5","Type":"ContainerStarted","Data":"c4b9d200428f494a905019dd15e68e4f823c1aac234b7b00cf2439d4e968573c"} Jul 07 18:36:42.972628 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:42.972592 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t248s/must-gather-vdjgj" event={"ID":"df76d70d-a2a1-4c36-85ed-4c21687888b5","Type":"ContainerStarted","Data":"1bb52888e848b22a50adfdc2034afa1655b27b2bd139ef4d0706f32a96b34a76"} Jul 07 18:36:42.973195 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:42.973135 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t248s/must-gather-vdjgj" event={"ID":"df76d70d-a2a1-4c36-85ed-4c21687888b5","Type":"ContainerStarted","Data":"3d15c3401d4c7f36d0c01bab20dd5a87448fa634e1baafdd7aae8afe02b9a49e"} Jul 07 18:36:44.083856 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:44.083814 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6s26g_1cd8e48b-8053-42a2-b2fb-777e20a75ed1/global-pull-secret-syncer/0.log" Jul 07 18:36:44.724350 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:44.724284 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9glvk_9f2fe74e-03fd-4704-a472-8965dd5d4230/konnectivity-agent/0.log" Jul 07 18:36:44.992737 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:44.992654 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-242.ec2.internal_79639d5bc5ecaca9c381236434449ff7/haproxy/0.log" Jul 07 18:36:49.303196 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:49.303162 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hs4gv_92e795c1-96eb-4080-91d7-5dfae785f503/kuadrant-console-plugin/0.log" Jul 07 18:36:49.715589 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:49.715429 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hzwz6_bb22ceb1-efb9-448e-acde-2b954c0f96e7/manager/0.log" Jul 07 18:36:52.171907 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:52.171862 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-xlgtt_c9bb7e59-91dc-4401-9e8a-a1d2bdf92c95/cluster-monitoring-operator/0.log" Jul 07 18:36:52.713776 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:52.713743 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qwk6l_b91fa988-036f-40d9-b1bf-6d62b311d082/node-exporter/0.log" Jul 07 18:36:52.752737 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:52.752706 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qwk6l_b91fa988-036f-40d9-b1bf-6d62b311d082/kube-rbac-proxy/0.log" Jul 07 18:36:52.796996 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:52.796967 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qwk6l_b91fa988-036f-40d9-b1bf-6d62b311d082/init-textfile/0.log" Jul 07 18:36:53.547453 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.547394 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t248s/must-gather-vdjgj" podStartSLOduration=12.900979629 podStartE2EDuration="13.547370632s" totalImagesPullingTime="646.391003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:36:43.004522219 +0000 UTC m=+2548.257638799" watchObservedRunningTime="2026-07-07 18:36:53.547370632 +0000 UTC m=+2558.800487213" Jul 07 18:36:53.549693 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.549660 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq"] Jul 07 18:36:53.555486 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.555459 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.580300 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.580263 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq"] Jul 07 18:36:53.716505 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.716465 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-sys\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.716690 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.716523 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-podres\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.716690 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.716570 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-proc\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.716690 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.716597 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjw96\" (UniqueName: \"kubernetes.io/projected/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-kube-api-access-rjw96\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.716820 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.716726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-lib-modules\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.818283 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.818235 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-proc\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.818462 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.818292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjw96\" (UniqueName: \"kubernetes.io/projected/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-kube-api-access-rjw96\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.818462 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.818380 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-proc\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.818462 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.818406 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-lib-modules\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.818462 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.818454 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-sys\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.818861 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.818492 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-podres\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.818861 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.818530 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-lib-modules\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.818861 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.818551 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-sys\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.818861 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.818646 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-podres\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.831289 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.831256 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjw96\" (UniqueName: \"kubernetes.io/projected/7ac6ba98-8449-4cd1-9050-ba4a9b467f60-kube-api-access-rjw96\") pod \"perf-node-gather-daemonset-nc4hq\" (UID: \"7ac6ba98-8449-4cd1-9050-ba4a9b467f60\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:53.868239 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:53.868202 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:54.045154 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:54.045119 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq"] Jul 07 18:36:54.048390 ip-10-0-137-242 kubenswrapper[2557]: W0707 18:36:54.048356 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7ac6ba98_8449_4cd1_9050_ba4a9b467f60.slice/crio-1a3f250c6e0c9b7b0a7c5e79e6de9e365581c26ca39eb0ea4bae3a58f270782e WatchSource:0}: Error finding container 1a3f250c6e0c9b7b0a7c5e79e6de9e365581c26ca39eb0ea4bae3a58f270782e: Status 404 returned error can't find the container with id 1a3f250c6e0c9b7b0a7c5e79e6de9e365581c26ca39eb0ea4bae3a58f270782e Jul 07 18:36:54.059777 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:54.059745 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" event={"ID":"7ac6ba98-8449-4cd1-9050-ba4a9b467f60","Type":"ContainerStarted","Data":"1a3f250c6e0c9b7b0a7c5e79e6de9e365581c26ca39eb0ea4bae3a58f270782e"} Jul 07 18:36:55.067035 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:55.066998 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" event={"ID":"7ac6ba98-8449-4cd1-9050-ba4a9b467f60","Type":"ContainerStarted","Data":"f0e970a69721008b52277323c147ad65f55d1873e807bc039269cf39f73dbd1f"} Jul 07 18:36:55.067538 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:55.067165 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:36:55.091716 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:55.091657 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" podStartSLOduration=2.091636704 podStartE2EDuration="2.091636704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:36:55.090648888 +0000 UTC m=+2560.343765495" watchObservedRunningTime="2026-07-07 18:36:55.091636704 +0000 UTC m=+2560.344753283" Jul 07 18:36:58.669450 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:58.669418 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xd72b_29ef67eb-8543-4716-8f68-fe90e8cbe171/dns/0.log" Jul 07 18:36:58.708661 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:58.708632 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xd72b_29ef67eb-8543-4716-8f68-fe90e8cbe171/kube-rbac-proxy/0.log" Jul 07 18:36:58.771508 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:58.771480 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52tkx_1fffc956-ac04-4a88-a353-a9aa99e3a8a3/dns-node-resolver/0.log" Jul 07 18:36:59.581218 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:36:59.581190 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rnnrw_bfcd6894-b47b-4ecd-9dc2-38235226086c/node-ca/0.log" Jul 07 18:37:00.949189 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:00.949152 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74ffcdc9c9-945sb_98e93bae-c22c-4bde-a2bf-dacfc26d91a8/kube-auth-proxy/0.log" Jul 07 18:37:01.082341 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:01.082295 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-nc4hq" Jul 07 18:37:02.312974 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:02.312945 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h8zpz_62bcc290-f56a-4710-b664-c6234385c9a2/serve-healthcheck-canary/0.log" Jul 07 18:37:03.011714 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:03.011678 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-x8khq_d64a4e93-2ba3-4bf5-8f28-c62796e2d05c/insights-operator/0.log" Jul 07 18:37:03.013773 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:03.013739 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-x8khq_d64a4e93-2ba3-4bf5-8f28-c62796e2d05c/insights-operator/1.log" Jul 07 18:37:03.258655 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:03.258622 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pqdtv_005a0572-6d40-4446-8df8-3dd9f7bd72ca/kube-rbac-proxy/0.log" Jul 07 18:37:03.310404 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:03.310352 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pqdtv_005a0572-6d40-4446-8df8-3dd9f7bd72ca/exporter/0.log" Jul 07 18:37:03.378707 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:03.378674 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pqdtv_005a0572-6d40-4446-8df8-3dd9f7bd72ca/extractor/0.log" Jul 07 18:37:04.625429 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:04.625372 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74ffcdc9c9-945sb_98e93bae-c22c-4bde-a2bf-dacfc26d91a8/kube-auth-proxy/0.log" Jul 07 18:37:06.334509 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:06.334476 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74ffcdc9c9-945sb_98e93bae-c22c-4bde-a2bf-dacfc26d91a8/kube-auth-proxy/0.log" Jul 07 18:37:06.984781 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:06.984747 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cd65fb98-8qqlw_a29d1b58-e125-452d-9c5b-9ca24a9066a0/manager/0.log" Jul 07 18:37:07.012797 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:07.012772 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cd65fb98-8qqlw_a29d1b58-e125-452d-9c5b-9ca24a9066a0/copy-manifests/0.log" Jul 07 18:37:08.601495 ip-10-0-137-242 kubenswrapper[2557]: I0707 18:37:08.601462 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58f6946598-l688g_0458d2b7-6c91-4a55-b012-4a1e265ca008/manager/0.log"