Jul 16 17:30:38.807193 ip-10-0-128-217 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 17:30:38.807205 ip-10-0-128-217 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 17:30:38.807215 ip-10-0-128-217 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 17:30:38.807565 ip-10-0-128-217 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 17:30:48.091268 ip-10-0-128-217 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 16 17:30:48.091283 ip-10-0-128-217 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot be496c79b16d4732a6565a304af29356 -- Jul 16 17:32:34.858184 ip-10-0-128-217 systemd[1]: Starting Kubernetes Kubelet... Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.166784 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170625 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170639 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170644 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170648 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170651 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170654 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170657 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:32:35.218405 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170660 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170663 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170665 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170668 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170670 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170673 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170675 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170678 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170681 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170683 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170686 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170688 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170690 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170700 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170703 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170705 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170708 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170711 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170713 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170716 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:32:35.219239 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170719 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170722 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170724 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170728 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170731 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170733 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170736 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170739 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170742 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170744 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170747 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170750 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170752 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170755 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170758 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170760 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170763 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170765 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170768 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170771 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:32:35.220450 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170774 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170776 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170779 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170781 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170783 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170786 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170788 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170791 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170793 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170796 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170798 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170801 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170804 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170806 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170809 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170812 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170815 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170817 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170819 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170822 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:32:35.221075 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170824 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170827 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170830 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170832 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170835 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170837 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170840 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170843 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170846 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170848 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170851 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170854 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170859 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170862 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170865 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170868 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170871 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170875 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170878 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:32:35.221895 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170881 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170883 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170899 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170902 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170905 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170908 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170911 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170914 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170916 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170919 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170922 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170924 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.170927 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172071 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172097 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172102 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172108 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172113 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172117 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172121 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:32:35.222443 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172125 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172129 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172133 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172137 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172142 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172147 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172151 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172160 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172163 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172168 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172172 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172176 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172180 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172184 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172188 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172192 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172196 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172200 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172204 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172212 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:32:35.223321 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172216 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172220 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172224 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172232 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172239 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172246 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172252 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172265 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172270 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172274 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172279 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172283 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172291 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172295 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172299 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172303 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172313 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172317 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172322 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172326 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:32:35.224349 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172330 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172337 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172341 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172345 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172354 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172358 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172363 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172367 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172371 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172375 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172380 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172384 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172388 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172392 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172396 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172403 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172411 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172416 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172420 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:32:35.224907 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172424 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172429 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172434 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172437 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172441 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172445 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172449 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172453 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172457 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172461 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172469 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172473 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172477 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172482 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172487 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172490 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172495 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172502 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172507 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172511 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:32:35.225385 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172516 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172524 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172529 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172533 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172542 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172546 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172550 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172555 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172560 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172566 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172571 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172576 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.172580 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173537 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173545 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173550 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173554 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173563 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173567 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173572 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:32:35.225859 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173576 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173579 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173584 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173588 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173592 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173597 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173601 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173606 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173610 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173614 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173623 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173627 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173631 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173919 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173947 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173951 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173955 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173959 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173962 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:32:35.226350 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173965 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173968 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173972 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173975 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173979 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173981 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173984 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173990 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173993 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173996 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.173999 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174002 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174005 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174008 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174011 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174015 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174021 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174026 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174032 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174035 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:32:35.226873 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174038 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174041 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174044 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174047 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174051 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174054 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174057 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174060 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174063 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174066 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174069 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174075 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174077 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174080 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174083 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174085 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174088 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174091 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174094 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174100 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174103 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:32:35.227371 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174105 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174108 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174112 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174118 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174122 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174124 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174127 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174131 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174135 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174138 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174142 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174145 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174148 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174151 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174154 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174159 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174164 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174167 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174172 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:32:35.227925 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174175 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174178 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174181 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174184 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174187 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174189 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174192 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174197 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174200 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174203 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174206 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174208 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.174212 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174307 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174318 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174325 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174330 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174335 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174338 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174350 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174355 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 17:32:35.228393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174361 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174365 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174368 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174372 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174375 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174378 2576 flags.go:64] FLAG: --cgroup-root="" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174381 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174385 2576 flags.go:64] FLAG: --client-ca-file="" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174388 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174394 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174399 2576 flags.go:64] FLAG: --cluster-domain="" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174402 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174405 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174409 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174412 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174416 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174420 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174429 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174435 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174438 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174441 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174444 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174447 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174451 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 17:32:35.228975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174455 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174459 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174464 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174467 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174470 2576 flags.go:64] FLAG: --enable-server="true" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174473 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174479 2576 flags.go:64] FLAG: --event-burst="100" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174482 2576 flags.go:64] FLAG: --event-qps="50" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174485 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174488 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174493 2576 flags.go:64] FLAG: --eviction-hard="" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174497 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174501 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174504 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174507 2576 flags.go:64] FLAG: --eviction-soft="" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174513 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174517 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174520 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174523 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174529 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174532 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174535 2576 flags.go:64] FLAG: --feature-gates="" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174539 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174542 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174546 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 17:32:35.229653 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174549 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174553 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174558 2576 flags.go:64] FLAG: --help="false" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174562 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-217.ec2.internal" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174565 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174568 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174571 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174574 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174578 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174581 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174588 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174594 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174597 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174600 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174603 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174606 2576 flags.go:64] FLAG: --kube-reserved="" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174609 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174612 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174615 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174621 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174623 2576 flags.go:64] FLAG: --lock-file="" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174626 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174631 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174634 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 17:32:35.230305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174640 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174643 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174648 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174651 2576 flags.go:64] FLAG: --logging-format="text" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174657 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174660 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174663 2576 flags.go:64] FLAG: --manifest-url="" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174666 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174671 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174675 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174680 2576 flags.go:64] FLAG: --max-pods="110" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174684 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174689 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174692 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174696 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174699 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174702 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174705 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174713 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174718 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174722 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174725 2576 flags.go:64] FLAG: --pod-cidr="" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174728 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 17:32:35.231030 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174734 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174737 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174740 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174743 2576 flags.go:64] FLAG: --port="10250" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174750 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174755 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04e5f01bcbbe58ca5" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174759 2576 flags.go:64] FLAG: --qos-reserved="" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174762 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174766 2576 flags.go:64] FLAG: --register-node="true" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174770 2576 flags.go:64] FLAG: --register-with-taints="" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174774 2576 flags.go:64] FLAG: --registry-burst="10" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174777 2576 flags.go:64] FLAG: --registry-qps="5" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174780 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174785 2576 flags.go:64] FLAG: --reserved-memory="" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174789 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174792 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174796 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174799 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174803 2576 flags.go:64] FLAG: --runonce="false" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174806 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174809 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174815 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174818 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174821 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174824 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174827 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 17:32:35.231582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174830 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174833 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174836 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174839 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174844 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174847 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174850 2576 flags.go:64] FLAG: --system-cgroups="" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174853 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174859 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174862 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174865 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174872 2576 flags.go:64] FLAG: --tls-min-version="" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174875 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174878 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174882 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174900 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174904 2576 flags.go:64] FLAG: --v="2" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174908 2576 flags.go:64] FLAG: --version="false" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174913 2576 flags.go:64] FLAG: --vmodule="" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174919 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.174923 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175139 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175143 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175146 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:32:35.232512 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175150 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175153 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175156 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175159 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175161 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175164 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175167 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175170 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175173 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175178 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175181 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175184 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175187 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175190 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175192 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175195 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175198 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175200 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175203 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175206 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:32:35.233426 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175209 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175214 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175219 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175221 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175224 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175235 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175239 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175242 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175245 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175248 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175251 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175254 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175257 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175263 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175266 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175268 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175271 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175274 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175276 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175279 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:32:35.234180 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175282 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175285 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175287 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175290 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175293 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175295 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175301 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175303 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175306 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175309 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175311 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175314 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175316 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175319 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175323 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175325 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175328 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175331 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175333 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175338 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:32:35.234685 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175341 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175344 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175347 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175350 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175354 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175356 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175359 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175362 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175364 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175367 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175371 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175378 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175381 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175384 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175386 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175389 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175392 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175395 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175398 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:32:35.235224 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175402 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175406 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175409 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175412 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175415 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175420 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175423 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175427 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175431 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175434 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175437 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175441 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175443 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175446 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175449 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175452 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:32:35.235692 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.175455 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.175999 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.183994 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.184012 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.184609 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.184616 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184674 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184680 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184683 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184686 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184689 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184692 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184695 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184697 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:32:35.236106 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184701 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184704 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184706 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184709 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184712 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184715 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184717 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184720 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184723 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184725 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184728 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184731 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184733 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184736 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184738 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184741 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184744 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184747 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184750 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184753 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:32:35.236448 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184755 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184758 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184761 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184763 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184766 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184768 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184771 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184773 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184776 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184779 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184782 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184785 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184787 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184789 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184792 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184794 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184799 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184803 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184806 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184810 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:32:35.236942 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184813 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184816 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184818 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184821 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184824 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184826 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184829 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184832 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184835 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184837 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184840 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184843 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184846 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184849 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184851 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184855 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184858 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184861 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184865 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:32:35.237413 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184868 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184871 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184873 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184876 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184879 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184881 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184884 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184900 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184903 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184905 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184908 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184910 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184913 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184916 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184918 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184921 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184923 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184926 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184929 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184932 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:32:35.237871 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184934 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184940 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184943 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184945 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184948 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184952 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184956 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184959 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184962 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184964 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184967 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.184969 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.184974 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185088 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185092 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:32:35.238363 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185095 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185098 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185101 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185104 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185106 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185109 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185111 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185115 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185117 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185120 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185122 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185125 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185127 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185130 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185132 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185135 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185138 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185140 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185145 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185148 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:32:35.238737 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185150 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185154 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185156 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185159 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185162 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185164 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185168 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185171 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185174 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185178 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185181 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185184 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185187 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185192 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185195 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185197 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185200 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185202 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185205 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:32:35.239242 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185207 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185210 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185212 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185215 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185217 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185220 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185222 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185225 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185227 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185230 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185232 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185236 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185239 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185241 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185244 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185247 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185249 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185252 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185254 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185257 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:32:35.239713 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185259 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185262 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185265 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185267 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185270 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185273 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185276 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185279 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185281 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185284 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185286 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185289 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185291 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185294 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185296 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185299 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185301 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185304 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185306 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185309 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:32:35.240247 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185311 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185313 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185316 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185320 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185323 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185325 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185328 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185330 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185334 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185337 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185339 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185341 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185345 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185349 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185352 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185355 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185358 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:32:35.240766 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:35.185360 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.185366 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.185475 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.188052 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.189195 2576 server.go:1019] "Starting client certificate rotation" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.191162 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.194309 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.194391 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.196298 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.209881 2576 log.go:25] "Validated CRI v1 runtime API" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.211986 2576 log.go:25] "Validated CRI v1 image API" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.213238 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 17:32:35.241533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.215224 2576 fs.go:135] Filesystem UUIDs: map[191bca17-e43e-46f3-a548-3b3f72f1807b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a1ff40b2-94ec-4aa9-8644-a16b10a14d9d:/dev/nvme0n1p4] Jul 16 17:32:35.242020 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.215245 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 17:32:35.241588 ip-10-0-128-217 systemd[1]: Started Kubernetes Kubelet. Jul 16 17:32:35.242240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.226038 2576 manager.go:216] Machine: {Timestamp:2026-07-16 17:32:35.225537791 +0000 UTC m=+0.267911775 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100964 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24d428f7d6edab0825ec4a04528067 SystemUUID:ec24d428-f7d6-edab-0825-ec4a04528067 BootID:be496c79-b16d-4732-a656-5a304af29356 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:35:1b:47:b1:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:35:1b:47:b1:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:6f:35:6a:d4:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 17:32:35.242240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.226141 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 17:32:35.242240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.226264 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 17:32:35.242240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.227128 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 17:32:35.242240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.227152 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-217.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 17:32:35.242240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.227772 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 17:32:35.242240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.227781 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 17:32:35.242240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.227788 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.227800 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.228946 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.228983 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.232588 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.232704 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.232731 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.232741 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.232754 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.232763 2576 kubelet.go:398] "Adding apiserver pod source" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.232771 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.233134 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.233150 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.234855 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.236416 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.236443 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.236572 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.236583 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.236589 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.236595 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.236602 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.236608 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.237027 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.237033 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.237039 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.237045 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.237053 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.237060 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.237587 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.237594 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.240631 2576 server.go:1268] "Started kubelet" Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.240696 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.240706 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.240771 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 17:32:35.242735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.241830 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 17:32:35.244273 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.243085 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 16 17:32:35.245228 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:35.245205 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 17:32:35.246020 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:35.245985 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 17:32:35.246160 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.246146 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-217.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 17:32:35.246223 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:35.246201 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 17:32:35.246638 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:35.245920 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-217.ec2.internal.18c2d581a824892a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-217.ec2.internal,UID:ip-10-0-128-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-217.ec2.internal,},FirstTimestamp:2026-07-16 17:32:35.240601898 +0000 UTC m=+0.282975882,LastTimestamp:2026-07-16 17:32:35.240601898 +0000 UTC m=+0.282975882,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-217.ec2.internal,}" Jul 16 17:32:35.248312 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.248296 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 17:32:35.248631 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.248614 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 17:32:35.251807 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:35.251789 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:35.252274 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.252262 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 17:32:35.252274 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.252274 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 17:32:35.252450 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.252440 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 17:32:35.252520 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.252511 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 16 17:32:35.252549 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.252520 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 16 17:32:35.253201 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.253185 2576 factory.go:55] Registering systemd factory Jul 16 17:32:35.253201 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.253203 2576 factory.go:223] Registration of the systemd container factory successfully Jul 16 17:32:35.253396 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.253385 2576 factory.go:153] Registering CRI-O factory Jul 16 17:32:35.253457 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.253399 2576 factory.go:223] Registration of the crio container factory successfully Jul 16 17:32:35.253457 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.253446 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 17:32:35.253544 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.253468 2576 factory.go:103] Registering Raw factory Jul 16 17:32:35.253544 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.253483 2576 manager.go:1196] Started watching for new ooms in manager Jul 16 17:32:35.253809 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.253795 2576 manager.go:349] Starting recovery of all containers Jul 16 17:32:35.254797 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.254775 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 17:32:35.256138 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:35.256093 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 17:32:35.256138 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:35.256125 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 17:32:35.266253 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.266235 2576 manager.go:354] Recovery completed Jul 16 17:32:35.271692 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.271675 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:32:35.273396 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.273374 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:32:35.273465 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.273405 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:32:35.273465 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.273420 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:32:35.274407 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.274394 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 17:32:35.274407 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.274404 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:32:35.274407 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.274407 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 17:32:35.274552 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.274425 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:32:35.275123 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.275092 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:32:35.275190 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.275131 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:32:35.275190 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:35.275142 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:32:35.275284 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:35.275223 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-217.ec2.internal.18c2d581aa18dfec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-217.ec2.internal,UID:ip-10-0-128-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-217.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-217.ec2.internal,},FirstTimestamp:2026-07-16 17:32:35.273392108 +0000 UTC m=+0.315766091,LastTimestamp:2026-07-16 17:32:35.273392108 +0000 UTC m=+0.315766091,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-217.ec2.internal,}" Jul 16 17:32:36.657443 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.657429 2576 policy_none.go:49] "None policy: Start" Jul 16 17:32:36.657482 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.657448 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 17:32:36.657482 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.657459 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 17:32:36.658224 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.658207 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4x24v" Jul 16 17:32:36.659373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.659363 2576 policy_none.go:47] "Start" Jul 16 17:32:36.664608 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.664591 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4x24v" Jul 16 17:32:36.666127 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.666041 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-217.ec2.internal.18c2d581aa192ee5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-217.ec2.internal,UID:ip-10-0-128-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-217.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-217.ec2.internal,},FirstTimestamp:2026-07-16 17:32:35.273412325 +0000 UTC m=+0.315786311,LastTimestamp:2026-07-16 17:32:35.273412325 +0000 UTC m=+0.315786311,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-217.ec2.internal,}" Jul 16 17:32:36.690408 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.690263 2576 manager.go:337] "Starting Device Plugin manager" Jul 16 17:32:36.690493 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.690440 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 17:32:36.690493 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.690456 2576 server.go:87] "Starting device plugin registration server" Jul 16 17:32:36.690688 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.690677 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 17:32:36.690750 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.690691 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 17:32:36.690835 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.690815 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 17:32:36.690928 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.690918 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 17:32:36.690928 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.690927 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 17:32:36.691376 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.691358 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 17:32:36.691442 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.691414 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:36.765111 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.765073 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 17:32:36.767249 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.765122 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 17:32:36.767249 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.765144 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 17:32:36.767249 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.765175 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 17:32:36.767920 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.767903 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:32:36.791196 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.791176 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:32:36.792108 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.792092 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:32:36.792190 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.792123 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:32:36.792190 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.792137 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:32:36.792190 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.792162 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-217.ec2.internal" Jul 16 17:32:36.803084 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.803063 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-217.ec2.internal" Jul 16 17:32:36.803132 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.803089 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-217.ec2.internal\": node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:36.831243 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.831217 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:36.866277 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.866219 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal"] Jul 16 17:32:36.866346 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.866287 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:32:36.867893 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.867877 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:32:36.867978 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.867907 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:32:36.867978 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.867917 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:32:36.870290 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.870278 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:32:36.870407 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.870393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal" Jul 16 17:32:36.870440 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.870425 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:32:36.871016 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.871001 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:32:36.871086 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.871028 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:32:36.871086 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.871039 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:32:36.872225 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.872210 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:32:36.872290 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.872239 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:32:36.872290 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.872249 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:32:36.873279 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.873261 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" Jul 16 17:32:36.873362 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.873289 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:32:36.874046 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.874033 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:32:36.874119 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.874056 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:32:36.874119 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.874069 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:32:36.903360 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.903340 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-217.ec2.internal\" not found" node="ip-10-0-128-217.ec2.internal" Jul 16 17:32:36.908092 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.908076 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-217.ec2.internal\" not found" node="ip-10-0-128-217.ec2.internal" Jul 16 17:32:36.932244 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:36.932224 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:36.934457 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.934438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f1735a04c5831a09a654d84791ec904-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal\" (UID: \"0f1735a04c5831a09a654d84791ec904\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" Jul 16 17:32:36.934518 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.934464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f1735a04c5831a09a654d84791ec904-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal\" (UID: \"0f1735a04c5831a09a654d84791ec904\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" Jul 16 17:32:36.934518 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:36.934482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc34ace512f7ad807781d2c531ab57e7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-217.ec2.internal\" (UID: \"cc34ace512f7ad807781d2c531ab57e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.032549 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:37.032511 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:37.034763 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.034742 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc34ace512f7ad807781d2c531ab57e7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-217.ec2.internal\" (UID: \"cc34ace512f7ad807781d2c531ab57e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.034837 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.034800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc34ace512f7ad807781d2c531ab57e7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-217.ec2.internal\" (UID: \"cc34ace512f7ad807781d2c531ab57e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.034891 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.034842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f1735a04c5831a09a654d84791ec904-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal\" (UID: \"0f1735a04c5831a09a654d84791ec904\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.034891 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.034875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f1735a04c5831a09a654d84791ec904-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal\" (UID: \"0f1735a04c5831a09a654d84791ec904\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.034891 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.034886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f1735a04c5831a09a654d84791ec904-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal\" (UID: \"0f1735a04c5831a09a654d84791ec904\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.035008 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.034922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f1735a04c5831a09a654d84791ec904-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal\" (UID: \"0f1735a04c5831a09a654d84791ec904\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.133252 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:37.133179 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:37.205613 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.205575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.210436 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.210417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.234179 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:37.234151 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:37.334651 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:37.334616 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:37.435136 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:37.435059 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:37.535531 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:37.535500 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:37.569845 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.569821 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 17:32:37.570012 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.569993 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 17:32:37.636596 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:37.636564 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:37.639978 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.639956 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:32:37.662362 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.662341 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:32:37.668395 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.668370 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:27:36 +0000 UTC" deadline="2028-04-18 05:58:31.825579146 +0000 UTC" Jul 16 17:32:37.668395 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.668393 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15396h25m54.15718809s" Jul 16 17:32:37.736910 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:37.736840 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:37.786040 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.786007 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:32:37.786966 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.786943 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:32:37.787068 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.786973 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:32:37.787068 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.786984 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:32:37.837974 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:37.837941 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-217.ec2.internal\" not found" Jul 16 17:32:37.924681 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.924652 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:32:37.933665 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.933646 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.942138 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.942119 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:32:37.942872 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.942859 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" Jul 16 17:32:37.956903 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.956885 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:32:37.962595 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:37.962569 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc34ace512f7ad807781d2c531ab57e7.slice/crio-e94c41c5e8cdf1d81588a1296108c6df2057816a185768db95760300fca93127 WatchSource:0}: Error finding container e94c41c5e8cdf1d81588a1296108c6df2057816a185768db95760300fca93127: Status 404 returned error can't find the container with id e94c41c5e8cdf1d81588a1296108c6df2057816a185768db95760300fca93127 Jul 16 17:32:37.966882 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:37.966867 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:32:38.615087 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.615048 2576 apiserver.go:52] "Watching apiserver" Jul 16 17:32:38.621256 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.621237 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 17:32:38.622484 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.622464 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-n2xdq","openshift-multus/network-metrics-daemon-dmrj5","openshift-network-diagnostics/network-check-target-b5qk9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf","openshift-cluster-node-tuning-operator/tuned-5z4tz","openshift-dns/node-resolver-dfr8c","openshift-multus/multus-additional-cni-plugins-czpbw","openshift-network-operator/iptables-alerter-lzhl7","openshift-ovn-kubernetes/ovnkube-node-n8ksg","kube-system/global-pull-secret-syncer-65qpf","kube-system/konnectivity-agent-qzrj4","kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal","openshift-image-registry/node-ca-sc2p8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal"] Jul 16 17:32:38.626148 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.626127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.628236 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.628218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:38.628350 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.628281 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:38.628531 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.628508 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 17:32:38.628623 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.628606 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 17:32:38.628682 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.628651 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 17:32:38.628682 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.628663 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 17:32:38.628803 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.628748 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 17:32:38.628928 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.628915 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-frhwv\"" Jul 16 17:32:38.630332 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.630315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.632163 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.632140 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 17:32:38.632555 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.632444 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 17:32:38.632555 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.632509 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 17:32:38.632678 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.632630 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2mngw\"" Jul 16 17:32:38.632805 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.632730 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 17:32:38.632897 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.632882 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.634985 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.634966 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:32:38.635078 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.635047 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.635205 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.635186 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tz2c9\"" Jul 16 17:32:38.635278 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.635235 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 17:32:38.635278 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.635271 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 17:32:38.636980 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.636964 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:32:38.637493 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.637285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.637992 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.637971 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 17:32:38.638855 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.638126 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tjzhr\"" Jul 16 17:32:38.640217 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.639881 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 17:32:38.640217 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.639972 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z8lwh\"" Jul 16 17:32:38.640217 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.640061 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 17:32:38.640434 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.640218 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 17:32:38.640434 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.640379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.641343 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-os-release\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.641438 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.641438 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmb7\" (UniqueName: \"kubernetes.io/projected/ec0ea45e-427b-42b5-a095-2d6e8a0ef43f-kube-api-access-5vmb7\") pod \"node-ca-sc2p8\" (UID: \"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f\") " pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.641438 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-var-lib-kubelet\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.641578 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-host\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.641578 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjvj\" (UniqueName: \"kubernetes.io/projected/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-kube-api-access-rgjvj\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.641578 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7906aa23-cf11-4360-b0d0-a2e3a580bffa-iptables-alerter-script\") pod \"iptables-alerter-lzhl7\" (UID: \"7906aa23-cf11-4360-b0d0-a2e3a580bffa\") " pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.641578 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2b62\" (UniqueName: \"kubernetes.io/projected/7906aa23-cf11-4360-b0d0-a2e3a580bffa-kube-api-access-k2b62\") pod \"iptables-alerter-lzhl7\" (UID: \"7906aa23-cf11-4360-b0d0-a2e3a580bffa\") " pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.641755 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-sysconfig\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.641755 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-kubernetes\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.641755 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-run\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.641755 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec0ea45e-427b-42b5-a095-2d6e8a0ef43f-host\") pod \"node-ca-sc2p8\" (UID: \"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f\") " pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.641755 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7906aa23-cf11-4360-b0d0-a2e3a580bffa-host-slash\") pod \"iptables-alerter-lzhl7\" (UID: \"7906aa23-cf11-4360-b0d0-a2e3a580bffa\") " pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.641987 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-sys\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.641987 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-system-cni-dir\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.641987 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.641987 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.641987 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-sysctl-d\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.641987 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-lib-modules\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.641987 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-cnibin\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.641987 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.641999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-modprobe-d\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7t28\" (UniqueName: \"kubernetes.io/projected/2de5f0b6-b269-4e83-9942-ac6382480435-kube-api-access-l7t28\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec0ea45e-427b-42b5-a095-2d6e8a0ef43f-serviceca\") pod \"node-ca-sc2p8\" (UID: \"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f\") " pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-systemd\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2de5f0b6-b269-4e83-9942-ac6382480435-tmp\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd44t\" (UniqueName: \"kubernetes.io/projected/8680dbdf-6918-4634-9412-e88acf7f528a-kube-api-access-rd44t\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-sysctl-conf\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2de5f0b6-b269-4e83-9942-ac6382480435-etc-tuned\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.642353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.642903 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642637 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 17:32:38.642903 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642793 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jfrh9\"" Jul 16 17:32:38.642903 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.642813 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 17:32:38.644200 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.644186 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 17:32:38.644462 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.644444 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:32:38.644578 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.644558 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rlj96\"" Jul 16 17:32:38.644788 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.644774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.646814 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.646801 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 17:32:38.647067 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.647041 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 17:32:38.647139 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.647050 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:38.647139 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.647094 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9d7w5\"" Jul 16 17:32:38.647139 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.647119 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 17:32:38.647299 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.647151 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:38.647544 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.647527 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 17:32:38.647608 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.647549 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 17:32:38.647608 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.647558 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 17:32:38.649353 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.649337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:32:38.651870 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.651849 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:38.651959 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.651905 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:38.652020 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.651995 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qlk7n\"" Jul 16 17:32:38.652120 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.652104 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 17:32:38.652268 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.652249 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 17:32:38.669935 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.669914 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-77rz9" Jul 16 17:32:38.678870 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.678854 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-77rz9" Jul 16 17:32:38.697164 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.697148 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:32:38.734165 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.734149 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 17:32:38.736082 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:38.736057 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1735a04c5831a09a654d84791ec904.slice/crio-392b27fddaabceb5d19de6d91405508ed6ef71a87a40227b71aaf7b62290a6c4 WatchSource:0}: Error finding container 392b27fddaabceb5d19de6d91405508ed6ef71a87a40227b71aaf7b62290a6c4: Status 404 returned error can't find the container with id 392b27fddaabceb5d19de6d91405508ed6ef71a87a40227b71aaf7b62290a6c4 Jul 16 17:32:38.742410 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:38.742499 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-modprobe-d\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.742499 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7t28\" (UniqueName: \"kubernetes.io/projected/2de5f0b6-b269-4e83-9942-ac6382480435-kube-api-access-l7t28\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.742499 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-cni-dir\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.742499 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-kubelet\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.742632 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-ovnkube-config\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.742632 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2de5f0b6-b269-4e83-9942-ac6382480435-tmp\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.742632 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.742548 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:38.742632 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742571 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-modprobe-d\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.742632 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-socket-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.742632 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-socket-dir-parent\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.742876 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-var-lib-cni-bin\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.742876 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-ovn-node-metrics-cert\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.742876 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-cnibin\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.742876 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.742775 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs podName:8680dbdf-6918-4634-9412-e88acf7f528a nodeName:}" failed. No retries permitted until 2026-07-16 17:32:39.242751128 +0000 UTC m=+2.904314754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs") pod "network-metrics-daemon-dmrj5" (UID: "8680dbdf-6918-4634-9412-e88acf7f528a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:38.742876 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742788 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 17:32:38.742876 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-var-lib-kubelet\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.742876 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-slash\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.742876 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-var-lib-openvswitch\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-run-netns\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-hostroot\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-os-release\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmb7\" (UniqueName: \"kubernetes.io/projected/ec0ea45e-427b-42b5-a095-2d6e8a0ef43f-kube-api-access-5vmb7\") pod \"node-ca-sc2p8\" (UID: \"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f\") " pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.742978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-registration-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-run-netns\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-cni-bin\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-os-release\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjvj\" (UniqueName: \"kubernetes.io/projected/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-kube-api-access-rgjvj\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7906aa23-cf11-4360-b0d0-a2e3a580bffa-iptables-alerter-script\") pod \"iptables-alerter-lzhl7\" (UID: \"7906aa23-cf11-4360-b0d0-a2e3a580bffa\") " pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.743168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-run\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4da5bf01-f439-4e51-8ef5-6828fee14726-hosts-file\") pod \"node-resolver-dfr8c\" (UID: \"4da5bf01-f439-4e51-8ef5-6828fee14726\") " pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d67n\" (UniqueName: \"kubernetes.io/projected/4da5bf01-f439-4e51-8ef5-6828fee14726-kube-api-access-8d67n\") pod \"node-resolver-dfr8c\" (UID: \"4da5bf01-f439-4e51-8ef5-6828fee14726\") " pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-etc-kubernetes\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpmzw\" (UniqueName: \"kubernetes.io/projected/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-kube-api-access-gpmzw\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec0ea45e-427b-42b5-a095-2d6e8a0ef43f-host\") pod \"node-ca-sc2p8\" (UID: \"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f\") " pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-run\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7906aa23-cf11-4360-b0d0-a2e3a580bffa-host-slash\") pod \"iptables-alerter-lzhl7\" (UID: \"7906aa23-cf11-4360-b0d0-a2e3a580bffa\") " pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743406 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7906aa23-cf11-4360-b0d0-a2e3a580bffa-host-slash\") pod \"iptables-alerter-lzhl7\" (UID: \"7906aa23-cf11-4360-b0d0-a2e3a580bffa\") " pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743407 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec0ea45e-427b-42b5-a095-2d6e8a0ef43f-host\") pod \"node-ca-sc2p8\" (UID: \"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f\") " pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-system-cni-dir\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-run-k8s-cni-cncf-io\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-ovnkube-script-lib\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743515 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.743684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-systemd-units\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-kubelet-config\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743580 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7906aa23-cf11-4360-b0d0-a2e3a580bffa-iptables-alerter-script\") pod \"iptables-alerter-lzhl7\" (UID: \"7906aa23-cf11-4360-b0d0-a2e3a580bffa\") " pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-lib-modules\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743768 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-sys-fs\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-daemon-config\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-run-systemd\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-env-overrides\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e9ae6c1-ca64-4d54-9283-3cc84e8300bf-agent-certs\") pod \"konnectivity-agent-qzrj4\" (UID: \"6e9ae6c1-ca64-4d54-9283-3cc84e8300bf\") " pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743898 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-lib-modules\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-node-log\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-dbus\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:38.744386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.743975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec0ea45e-427b-42b5-a095-2d6e8a0ef43f-serviceca\") pod \"node-ca-sc2p8\" (UID: \"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f\") " pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-systemd\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-run-openvswitch\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744136 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-systemd\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-log-socket\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd44t\" (UniqueName: \"kubernetes.io/projected/8680dbdf-6918-4634-9412-e88acf7f528a-kube-api-access-rd44t\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-sysctl-conf\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2de5f0b6-b269-4e83-9942-ac6382480435-etc-tuned\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-os-release\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-etc-openvswitch\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.745056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ae6c1-ca64-4d54-9283-3cc84e8300bf-konnectivity-ca\") pod \"konnectivity-agent-qzrj4\" (UID: \"6e9ae6c1-ca64-4d54-9283-3cc84e8300bf\") " pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-var-lib-kubelet\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-host\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da5bf01-f439-4e51-8ef5-6828fee14726-tmp-dir\") pod \"node-resolver-dfr8c\" (UID: \"4da5bf01-f439-4e51-8ef5-6828fee14726\") " pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-cni-binary-copy\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-sysctl-conf\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744587 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-host\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2b62\" (UniqueName: \"kubernetes.io/projected/7906aa23-cf11-4360-b0d0-a2e3a580bffa-kube-api-access-k2b62\") pod \"iptables-alerter-lzhl7\" (UID: \"7906aa23-cf11-4360-b0d0-a2e3a580bffa\") " pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744636 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744654 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-var-lib-kubelet\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-sysconfig\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745461 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744692 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-kubernetes\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-run-multus-certs\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-run-ovn\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-sysconfig\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-sys\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-conf-dir\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744824 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-kubernetes\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744834 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5bd\" (UniqueName: \"kubernetes.io/projected/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-kube-api-access-kg5bd\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-sys\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-system-cni-dir\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-var-lib-cni-multus\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-system-cni-dir\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.745901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-run-ovn-kubernetes\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.746292 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-sysctl-d\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.746292 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-device-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.746292 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.744993 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec0ea45e-427b-42b5-a095-2d6e8a0ef43f-serviceca\") pod \"node-ca-sc2p8\" (UID: \"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f\") " pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.746292 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.745010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvnj\" (UniqueName: \"kubernetes.io/projected/df3b7576-550d-40c1-b516-c449e21acaca-kube-api-access-qtvnj\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.746292 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.745037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-cni-netd\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.746292 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.745044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2de5f0b6-b269-4e83-9942-ac6382480435-etc-sysctl-d\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.746292 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.745065 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-cnibin\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.746292 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.745129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-cnibin\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.746506 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.746355 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2de5f0b6-b269-4e83-9942-ac6382480435-tmp\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.746506 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.746417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2de5f0b6-b269-4e83-9942-ac6382480435-etc-tuned\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.754116 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.754086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjvj\" (UniqueName: \"kubernetes.io/projected/3463dcd4-e95b-46ee-b1a0-a1c242cc91ef-kube-api-access-rgjvj\") pod \"multus-additional-cni-plugins-czpbw\" (UID: \"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef\") " pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.754116 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.754102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmb7\" (UniqueName: \"kubernetes.io/projected/ec0ea45e-427b-42b5-a095-2d6e8a0ef43f-kube-api-access-5vmb7\") pod \"node-ca-sc2p8\" (UID: \"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f\") " pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.754266 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.754092 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2b62\" (UniqueName: \"kubernetes.io/projected/7906aa23-cf11-4360-b0d0-a2e3a580bffa-kube-api-access-k2b62\") pod \"iptables-alerter-lzhl7\" (UID: \"7906aa23-cf11-4360-b0d0-a2e3a580bffa\") " pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.754505 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.754489 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7t28\" (UniqueName: \"kubernetes.io/projected/2de5f0b6-b269-4e83-9942-ac6382480435-kube-api-access-l7t28\") pod \"tuned-5z4tz\" (UID: \"2de5f0b6-b269-4e83-9942-ac6382480435\") " pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.754552 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.754536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd44t\" (UniqueName: \"kubernetes.io/projected/8680dbdf-6918-4634-9412-e88acf7f528a-kube-api-access-rd44t\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:38.768692 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.768646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" event={"ID":"0f1735a04c5831a09a654d84791ec904","Type":"ContainerStarted","Data":"392b27fddaabceb5d19de6d91405508ed6ef71a87a40227b71aaf7b62290a6c4"} Jul 16 17:32:38.769625 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.769604 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal" event={"ID":"cc34ace512f7ad807781d2c531ab57e7","Type":"ContainerStarted","Data":"e94c41c5e8cdf1d81588a1296108c6df2057816a185768db95760300fca93127"} Jul 16 17:32:38.846189 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-system-cni-dir\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.846342 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-run-k8s-cni-cncf-io\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.846342 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-ovnkube-script-lib\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846342 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.846342 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-systemd-units\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846342 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-kubelet-config\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:38.846342 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-system-cni-dir\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.846342 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-sys-fs\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-daemon-config\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-systemd-units\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-run-systemd\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-env-overrides\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-run-systemd\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e9ae6c1-ca64-4d54-9283-3cc84e8300bf-agent-certs\") pod \"konnectivity-agent-qzrj4\" (UID: \"6e9ae6c1-ca64-4d54-9283-3cc84e8300bf\") " pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-node-log\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-dbus\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846498 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-kubelet-config\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-run-openvswitch\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-log-socket\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-run-k8s-cni-cncf-io\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.846714 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-os-release\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-etc-openvswitch\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-os-release\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ae6c1-ca64-4d54-9283-3cc84e8300bf-konnectivity-ca\") pod \"konnectivity-agent-qzrj4\" (UID: \"6e9ae6c1-ca64-4d54-9283-3cc84e8300bf\") " pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da5bf01-f439-4e51-8ef5-6828fee14726-tmp-dir\") pod \"node-resolver-dfr8c\" (UID: \"4da5bf01-f439-4e51-8ef5-6828fee14726\") " pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-cni-binary-copy\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-run-multus-certs\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-run-ovn\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-conf-dir\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5bd\" (UniqueName: \"kubernetes.io/projected/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-kube-api-access-kg5bd\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-var-lib-cni-multus\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.847347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-run-ovn-kubernetes\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-device-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvnj\" (UniqueName: \"kubernetes.io/projected/df3b7576-550d-40c1-b516-c449e21acaca-kube-api-access-qtvnj\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-cni-netd\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847066 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-log-socket\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-cni-dir\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-node-log\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-kubelet\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-ovnkube-config\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847149 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-kubelet\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-socket-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847178 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-env-overrides\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-socket-dir-parent\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-var-lib-cni-bin\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848025 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-ovn-node-metrics-cert\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-run-openvswitch\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-cnibin\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-var-lib-kubelet\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-slash\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-var-lib-openvswitch\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847301 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-socket-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-run-netns\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847330 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-dbus\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-hostroot\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-run-netns\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-socket-dir-parent\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846537 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-sys-fs\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-registration-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-var-lib-cni-bin\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.848635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-run-netns\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-cni-bin\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4da5bf01-f439-4e51-8ef5-6828fee14726-hosts-file\") pod \"node-resolver-dfr8c\" (UID: \"4da5bf01-f439-4e51-8ef5-6828fee14726\") " pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d67n\" (UniqueName: \"kubernetes.io/projected/4da5bf01-f439-4e51-8ef5-6828fee14726-kube-api-access-8d67n\") pod \"node-resolver-dfr8c\" (UID: \"4da5bf01-f439-4e51-8ef5-6828fee14726\") " pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.847495 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-etc-kubernetes\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmzw\" (UniqueName: \"kubernetes.io/projected/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-kube-api-access-gpmzw\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847572 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-daemon-config\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-cni-binary-copy\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-hostroot\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-var-lib-kubelet\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-cnibin\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-etc-openvswitch\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847692 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-slash\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.849321 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847729 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-var-lib-openvswitch\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-var-lib-cni-multus\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ae6c1-ca64-4d54-9283-3cc84e8300bf-konnectivity-ca\") pod \"konnectivity-agent-qzrj4\" (UID: \"6e9ae6c1-ca64-4d54-9283-3cc84e8300bf\") " pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-run-ovn-kubernetes\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847798 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-run-netns\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-device-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4da5bf01-f439-4e51-8ef5-6828fee14726-hosts-file\") pod \"node-resolver-dfr8c\" (UID: \"4da5bf01-f439-4e51-8ef5-6828fee14726\") " pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-cni-bin\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-host-run-multus-certs\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df3b7576-550d-40c1-b516-c449e21acaca-registration-dir\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-conf-dir\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.846867 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-ovnkube-script-lib\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847879 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-host-cni-netd\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-multus-cni-dir\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.850060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-run-ovn\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.850814 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.847607 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret podName:cb2e0f7c-1598-4d6f-8515-07473e51b8c9 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:39.347589375 +0000 UTC m=+3.009153002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret") pod "global-pull-secret-syncer-65qpf" (UID: "cb2e0f7c-1598-4d6f-8515-07473e51b8c9") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:38.850814 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.847997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da5bf01-f439-4e51-8ef5-6828fee14726-tmp-dir\") pod \"node-resolver-dfr8c\" (UID: \"4da5bf01-f439-4e51-8ef5-6828fee14726\") " pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.850814 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.848013 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-etc-kubernetes\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.850814 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.848600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-ovnkube-config\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.850814 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.849470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e9ae6c1-ca64-4d54-9283-3cc84e8300bf-agent-certs\") pod \"konnectivity-agent-qzrj4\" (UID: \"6e9ae6c1-ca64-4d54-9283-3cc84e8300bf\") " pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:32:38.850814 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.849832 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-ovn-node-metrics-cert\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.858408 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.858384 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:32:38.858408 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.858409 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:32:38.858591 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.858421 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rjrsw for pod openshift-network-diagnostics/network-check-target-b5qk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:38.858591 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:38.858477 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw podName:a4d28a25-bf42-4815-9387-e010faaa08d2 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:39.3584645 +0000 UTC m=+3.020028133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rjrsw" (UniqueName: "kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw") pod "network-check-target-b5qk9" (UID: "a4d28a25-bf42-4815-9387-e010faaa08d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:38.859566 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.859473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvnj\" (UniqueName: \"kubernetes.io/projected/df3b7576-550d-40c1-b516-c449e21acaca-kube-api-access-qtvnj\") pod \"aws-ebs-csi-driver-node-dcztf\" (UID: \"df3b7576-550d-40c1-b516-c449e21acaca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.860936 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.860913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d67n\" (UniqueName: \"kubernetes.io/projected/4da5bf01-f439-4e51-8ef5-6828fee14726-kube-api-access-8d67n\") pod \"node-resolver-dfr8c\" (UID: \"4da5bf01-f439-4e51-8ef5-6828fee14726\") " pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.861573 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.861548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmzw\" (UniqueName: \"kubernetes.io/projected/0e4227b6-a431-4403-98c3-f70f2a1d0c7a-kube-api-access-gpmzw\") pod \"multus-n2xdq\" (UID: \"0e4227b6-a431-4403-98c3-f70f2a1d0c7a\") " pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.861985 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.861952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5bd\" (UniqueName: \"kubernetes.io/projected/3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607-kube-api-access-kg5bd\") pod \"ovnkube-node-n8ksg\" (UID: \"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.936393 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.936302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-czpbw" Jul 16 17:32:38.943614 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.943592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sc2p8" Jul 16 17:32:38.946954 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:38.946924 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3463dcd4_e95b_46ee_b1a0_a1c242cc91ef.slice/crio-b5b04b74a7a25db3b371a616c41bc3d5cc521dc00f99c7e0123363a52ef654f8 WatchSource:0}: Error finding container b5b04b74a7a25db3b371a616c41bc3d5cc521dc00f99c7e0123363a52ef654f8: Status 404 returned error can't find the container with id b5b04b74a7a25db3b371a616c41bc3d5cc521dc00f99c7e0123363a52ef654f8 Jul 16 17:32:38.950928 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.950904 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lzhl7" Jul 16 17:32:38.952779 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:38.952757 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0ea45e_427b_42b5_a095_2d6e8a0ef43f.slice/crio-c7dc33a2db8952e0a84e63ae03848044df2a16918cf724e82034d46edc5410be WatchSource:0}: Error finding container c7dc33a2db8952e0a84e63ae03848044df2a16918cf724e82034d46edc5410be: Status 404 returned error can't find the container with id c7dc33a2db8952e0a84e63ae03848044df2a16918cf724e82034d46edc5410be Jul 16 17:32:38.955424 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.955401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" Jul 16 17:32:38.961285 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.961264 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" Jul 16 17:32:38.963004 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:38.962987 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7906aa23_cf11_4360_b0d0_a2e3a580bffa.slice/crio-0a96396d177f32dc332c72ff12e287cda083b4e80cfe648527c85e63b623d5f9 WatchSource:0}: Error finding container 0a96396d177f32dc332c72ff12e287cda083b4e80cfe648527c85e63b623d5f9: Status 404 returned error can't find the container with id 0a96396d177f32dc332c72ff12e287cda083b4e80cfe648527c85e63b623d5f9 Jul 16 17:32:38.966082 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:38.965981 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de5f0b6_b269_4e83_9942_ac6382480435.slice/crio-aa83078d9c5dd2e72156760c37f96d58a0a9957c6ca6f740dd0eb90b6f0a0bf2 WatchSource:0}: Error finding container aa83078d9c5dd2e72156760c37f96d58a0a9957c6ca6f740dd0eb90b6f0a0bf2: Status 404 returned error can't find the container with id aa83078d9c5dd2e72156760c37f96d58a0a9957c6ca6f740dd0eb90b6f0a0bf2 Jul 16 17:32:38.966669 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.966652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dfr8c" Jul 16 17:32:38.971903 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:38.971804 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3b7576_550d_40c1_b516_c449e21acaca.slice/crio-15f37f153092fd882555e83f1dbc31f1d35f26ba1a96ad81c54adbe677c8cdbd WatchSource:0}: Error finding container 15f37f153092fd882555e83f1dbc31f1d35f26ba1a96ad81c54adbe677c8cdbd: Status 404 returned error can't find the container with id 15f37f153092fd882555e83f1dbc31f1d35f26ba1a96ad81c54adbe677c8cdbd Jul 16 17:32:38.972005 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.971986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n2xdq" Jul 16 17:32:38.977980 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.977958 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:32:38.981259 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:38.981210 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da5bf01_f439_4e51_8ef5_6828fee14726.slice/crio-167a2fb442d47a78cddd0c03d967fc64ad4130634abcc75658eea6c167f336c8 WatchSource:0}: Error finding container 167a2fb442d47a78cddd0c03d967fc64ad4130634abcc75658eea6c167f336c8: Status 404 returned error can't find the container with id 167a2fb442d47a78cddd0c03d967fc64ad4130634abcc75658eea6c167f336c8 Jul 16 17:32:38.982036 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:38.981891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:32:38.994173 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:38.994150 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce7cd9b_b3ce_4fd5_83ba_9fbff3f7a607.slice/crio-1890331f171f0650f88e6f1d35d6bd8d11519e4a9110cd423ba5f41f5a1a0afe WatchSource:0}: Error finding container 1890331f171f0650f88e6f1d35d6bd8d11519e4a9110cd423ba5f41f5a1a0afe: Status 404 returned error can't find the container with id 1890331f171f0650f88e6f1d35d6bd8d11519e4a9110cd423ba5f41f5a1a0afe Jul 16 17:32:38.995946 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:32:38.995897 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9ae6c1_ca64_4d54_9283_3cc84e8300bf.slice/crio-8a53a90ee3a6497bcec5ef391a6d86deda0ba11e8a0095748ab47dd973f927e6 WatchSource:0}: Error finding container 8a53a90ee3a6497bcec5ef391a6d86deda0ba11e8a0095748ab47dd973f927e6: Status 404 returned error can't find the container with id 8a53a90ee3a6497bcec5ef391a6d86deda0ba11e8a0095748ab47dd973f927e6 Jul 16 17:32:39.249910 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.249832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:39.250060 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:39.249964 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:39.250060 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:39.250019 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs podName:8680dbdf-6918-4634-9412-e88acf7f528a nodeName:}" failed. No retries permitted until 2026-07-16 17:32:40.250005312 +0000 UTC m=+3.911568922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs") pod "network-metrics-daemon-dmrj5" (UID: "8680dbdf-6918-4634-9412-e88acf7f528a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:39.350589 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.350540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:39.350773 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:39.350746 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:39.350846 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:39.350825 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret podName:cb2e0f7c-1598-4d6f-8515-07473e51b8c9 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:40.350802081 +0000 UTC m=+4.012365715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret") pod "global-pull-secret-syncer-65qpf" (UID: "cb2e0f7c-1598-4d6f-8515-07473e51b8c9") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:39.451033 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.450997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:39.451206 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:39.451186 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:32:39.451264 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:39.451212 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:32:39.451264 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:39.451227 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rjrsw for pod openshift-network-diagnostics/network-check-target-b5qk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:39.451357 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:39.451286 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw podName:a4d28a25-bf42-4815-9387-e010faaa08d2 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:40.451269936 +0000 UTC m=+4.112833547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rjrsw" (UniqueName: "kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw") pod "network-check-target-b5qk9" (UID: "a4d28a25-bf42-4815-9387-e010faaa08d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:39.682105 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.681995 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:27:38 +0000 UTC" deadline="2028-02-01 03:01:16.873486418 +0000 UTC" Jul 16 17:32:39.682105 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.682061 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13545h28m37.191429271s" Jul 16 17:32:39.774127 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.774023 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzrj4" event={"ID":"6e9ae6c1-ca64-4d54-9283-3cc84e8300bf","Type":"ContainerStarted","Data":"8a53a90ee3a6497bcec5ef391a6d86deda0ba11e8a0095748ab47dd973f927e6"} Jul 16 17:32:39.775932 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.775773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerStarted","Data":"1890331f171f0650f88e6f1d35d6bd8d11519e4a9110cd423ba5f41f5a1a0afe"} Jul 16 17:32:39.777272 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.777165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dfr8c" event={"ID":"4da5bf01-f439-4e51-8ef5-6828fee14726","Type":"ContainerStarted","Data":"167a2fb442d47a78cddd0c03d967fc64ad4130634abcc75658eea6c167f336c8"} Jul 16 17:32:39.779389 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.779360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" event={"ID":"df3b7576-550d-40c1-b516-c449e21acaca","Type":"ContainerStarted","Data":"15f37f153092fd882555e83f1dbc31f1d35f26ba1a96ad81c54adbe677c8cdbd"} Jul 16 17:32:39.781355 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.781291 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" event={"ID":"2de5f0b6-b269-4e83-9942-ac6382480435","Type":"ContainerStarted","Data":"aa83078d9c5dd2e72156760c37f96d58a0a9957c6ca6f740dd0eb90b6f0a0bf2"} Jul 16 17:32:39.783753 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.783668 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sc2p8" event={"ID":"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f","Type":"ContainerStarted","Data":"c7dc33a2db8952e0a84e63ae03848044df2a16918cf724e82034d46edc5410be"} Jul 16 17:32:39.786028 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.786005 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n2xdq" event={"ID":"0e4227b6-a431-4403-98c3-f70f2a1d0c7a","Type":"ContainerStarted","Data":"4ff7eb1baad6fe1c18383c0042eec79b42da99000ecccb94b44e7aeeed1e42c2"} Jul 16 17:32:39.788276 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.788250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lzhl7" event={"ID":"7906aa23-cf11-4360-b0d0-a2e3a580bffa","Type":"ContainerStarted","Data":"0a96396d177f32dc332c72ff12e287cda083b4e80cfe648527c85e63b623d5f9"} Jul 16 17:32:39.790198 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:39.790143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerStarted","Data":"b5b04b74a7a25db3b371a616c41bc3d5cc521dc00f99c7e0123363a52ef654f8"} Jul 16 17:32:40.258811 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:40.258665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:40.259041 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.259015 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:40.259114 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.259086 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs podName:8680dbdf-6918-4634-9412-e88acf7f528a nodeName:}" failed. No retries permitted until 2026-07-16 17:32:42.259067557 +0000 UTC m=+5.920631171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs") pod "network-metrics-daemon-dmrj5" (UID: "8680dbdf-6918-4634-9412-e88acf7f528a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:40.359518 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:40.359268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:40.359736 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.359715 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:40.359812 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.359785 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret podName:cb2e0f7c-1598-4d6f-8515-07473e51b8c9 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:42.359767122 +0000 UTC m=+6.021330736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret") pod "global-pull-secret-syncer-65qpf" (UID: "cb2e0f7c-1598-4d6f-8515-07473e51b8c9") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:40.460597 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:40.460560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:40.460786 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.460767 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:32:40.460861 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.460787 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:32:40.460861 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.460814 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rjrsw for pod openshift-network-diagnostics/network-check-target-b5qk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:40.460966 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.460872 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw podName:a4d28a25-bf42-4815-9387-e010faaa08d2 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:42.460852649 +0000 UTC m=+6.122416265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rjrsw" (UniqueName: "kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw") pod "network-check-target-b5qk9" (UID: "a4d28a25-bf42-4815-9387-e010faaa08d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:40.682408 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:40.682318 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:27:38 +0000 UTC" deadline="2028-02-01 07:51:59.341861772 +0000 UTC" Jul 16 17:32:40.682408 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:40.682361 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13550h19m18.659506313s" Jul 16 17:32:40.765843 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:40.765814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:40.765843 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:40.765814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:40.766067 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.765955 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:40.766067 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:40.765966 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:40.766067 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.766024 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:40.766223 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:40.766096 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:40.805899 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:40.805864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal" event={"ID":"cc34ace512f7ad807781d2c531ab57e7","Type":"ContainerStarted","Data":"3d87e4dff679edd4d04af2de8114adf1459e33221b3ac19e0cd3f929d93a48ca"} Jul 16 17:32:42.279325 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:42.278590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:42.279325 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.278815 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:42.279325 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.278890 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs podName:8680dbdf-6918-4634-9412-e88acf7f528a nodeName:}" failed. No retries permitted until 2026-07-16 17:32:46.2788646 +0000 UTC m=+9.940428214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs") pod "network-metrics-daemon-dmrj5" (UID: "8680dbdf-6918-4634-9412-e88acf7f528a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:42.380507 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:42.379890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:42.380507 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.380031 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:42.380507 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.380096 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret podName:cb2e0f7c-1598-4d6f-8515-07473e51b8c9 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:46.380078521 +0000 UTC m=+10.041642151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret") pod "global-pull-secret-syncer-65qpf" (UID: "cb2e0f7c-1598-4d6f-8515-07473e51b8c9") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:42.481027 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:42.480989 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:42.481212 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.481185 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:32:42.481212 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.481210 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:32:42.481366 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.481223 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rjrsw for pod openshift-network-diagnostics/network-check-target-b5qk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:42.481366 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.481278 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw podName:a4d28a25-bf42-4815-9387-e010faaa08d2 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:46.481261981 +0000 UTC m=+10.142825618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rjrsw" (UniqueName: "kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw") pod "network-check-target-b5qk9" (UID: "a4d28a25-bf42-4815-9387-e010faaa08d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:42.766317 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:42.766163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:42.766317 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:42.766172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:42.766532 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.766344 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:42.766532 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:42.766217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:42.766532 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.766435 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:42.766668 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:42.766550 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:44.766053 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:44.765772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:44.766053 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:44.765817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:44.766053 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:44.765780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:44.766053 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:44.765907 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:44.766053 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:44.766024 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:44.766695 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:44.766103 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:46.314747 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:46.314718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:46.315225 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.314849 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:46.315225 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.314922 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs podName:8680dbdf-6918-4634-9412-e88acf7f528a nodeName:}" failed. No retries permitted until 2026-07-16 17:32:54.314901871 +0000 UTC m=+17.976465488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs") pod "network-metrics-daemon-dmrj5" (UID: "8680dbdf-6918-4634-9412-e88acf7f528a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:46.415548 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:46.415499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:46.415744 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.415635 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:46.415744 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.415712 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret podName:cb2e0f7c-1598-4d6f-8515-07473e51b8c9 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:54.415680907 +0000 UTC m=+18.077244522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret") pod "global-pull-secret-syncer-65qpf" (UID: "cb2e0f7c-1598-4d6f-8515-07473e51b8c9") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:46.516960 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:46.516875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:46.517162 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.517063 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:32:46.517162 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.517091 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:32:46.517162 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.517105 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rjrsw for pod openshift-network-diagnostics/network-check-target-b5qk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:46.517320 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.517166 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw podName:a4d28a25-bf42-4815-9387-e010faaa08d2 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:54.517146187 +0000 UTC m=+18.178709811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rjrsw" (UniqueName: "kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw") pod "network-check-target-b5qk9" (UID: "a4d28a25-bf42-4815-9387-e010faaa08d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:46.766527 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:46.766369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:46.766527 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:46.766491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:46.766527 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.766507 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:46.766812 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.766603 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:46.766812 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:46.766652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:46.766812 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:46.766714 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:48.766375 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:48.766340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:48.766904 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:48.766340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:48.766904 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:48.766495 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:48.766904 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:48.766351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:48.766904 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:48.766523 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:48.766904 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:48.766615 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:50.766246 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:50.766147 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:50.766246 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:50.766207 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:50.766721 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:50.766152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:50.766721 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:50.766309 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:50.766721 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:50.766425 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:50.766721 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:50.766512 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:52.765872 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:52.765837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:52.766333 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:52.765837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:52.766333 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:52.765971 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:52.766333 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:52.766058 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:52.766333 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:52.765845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:52.766333 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:52.766159 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:54.373593 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:54.373557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:54.374052 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.373668 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:54.374052 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.373733 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs podName:8680dbdf-6918-4634-9412-e88acf7f528a nodeName:}" failed. No retries permitted until 2026-07-16 17:33:10.373716999 +0000 UTC m=+34.035280612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs") pod "network-metrics-daemon-dmrj5" (UID: "8680dbdf-6918-4634-9412-e88acf7f528a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:32:54.474299 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:54.474265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:54.474426 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.474371 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:54.474426 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.474426 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret podName:cb2e0f7c-1598-4d6f-8515-07473e51b8c9 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:10.474412262 +0000 UTC m=+34.135975873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret") pod "global-pull-secret-syncer-65qpf" (UID: "cb2e0f7c-1598-4d6f-8515-07473e51b8c9") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:32:54.575550 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:54.575513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:54.575739 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.575692 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:32:54.575739 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.575733 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:32:54.575854 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.575747 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rjrsw for pod openshift-network-diagnostics/network-check-target-b5qk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:54.575854 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.575813 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw podName:a4d28a25-bf42-4815-9387-e010faaa08d2 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:10.575794468 +0000 UTC m=+34.237358093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rjrsw" (UniqueName: "kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw") pod "network-check-target-b5qk9" (UID: "a4d28a25-bf42-4815-9387-e010faaa08d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:32:54.765998 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:54.765966 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:54.766191 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:54.765969 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:54.766191 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.766096 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:54.766191 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:54.765969 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:54.766322 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.766189 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:54.766869 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:54.766833 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:56.766417 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:56.766382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:56.766856 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:56.766468 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:56.766856 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:56.766510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:56.766856 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:56.766519 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:56.766856 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:56.766594 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:56.766856 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:56.766679 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:58.766130 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:58.766096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:32:58.766583 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:58.766096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:32:58.766583 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:58.766216 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:32:58.766583 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:58.766331 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:32:58.766583 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:58.766105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:32:58.766583 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:32:58.766425 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:32:59.844306 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.844155 2576 generic.go:356] "Generic (PLEG): container finished" podID="3463dcd4-e95b-46ee-b1a0-a1c242cc91ef" containerID="1db6f54dab28eb9d6a25caa666b16a8d604aa37c24daf177e2cf886a58ff4840" exitCode=0 Jul 16 17:32:59.845094 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.844193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerDied","Data":"1db6f54dab28eb9d6a25caa666b16a8d604aa37c24daf177e2cf886a58ff4840"} Jul 16 17:32:59.845846 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.845825 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzrj4" event={"ID":"6e9ae6c1-ca64-4d54-9283-3cc84e8300bf","Type":"ContainerStarted","Data":"130f143ad3741472f512785587b902c7a9a93e38f57b23d517df8be778bb85a9"} Jul 16 17:32:59.847791 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.847773 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:32:59.848127 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.848108 2576 generic.go:356] "Generic (PLEG): container finished" podID="3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607" containerID="ec2c89ac6d811936f6203b2c4e6e6c8ecfe1ea5119d68616b5cc52442e337c7f" exitCode=1 Jul 16 17:32:59.848223 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.848169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerStarted","Data":"6dfa727398f98cdc54e3ac27014907b64230c44ac328930cbfdbbbd7774f5bed"} Jul 16 17:32:59.848223 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.848218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerDied","Data":"ec2c89ac6d811936f6203b2c4e6e6c8ecfe1ea5119d68616b5cc52442e337c7f"} Jul 16 17:32:59.848372 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.848261 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerStarted","Data":"8b5042c5447d4975c12c879b5638ded11a8581c336714cb6a3070345d0e55c05"} Jul 16 17:32:59.849692 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.849667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dfr8c" event={"ID":"4da5bf01-f439-4e51-8ef5-6828fee14726","Type":"ContainerStarted","Data":"ba3ea1e8d9e659b4977ecca369a03e35d647ef237a6b13b169feae73d7514da7"} Jul 16 17:32:59.851179 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.851159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" event={"ID":"df3b7576-550d-40c1-b516-c449e21acaca","Type":"ContainerStarted","Data":"266cace568dc70a2368e2e99bab8c1004a51263ed496f9dd4f76a6bc23381025"} Jul 16 17:32:59.852540 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.852518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" event={"ID":"2de5f0b6-b269-4e83-9942-ac6382480435","Type":"ContainerStarted","Data":"43df326c159c491f2f8f16ad3540f63a7caf53a062dc094f75130a1b8e4d8eb0"} Jul 16 17:32:59.853893 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.853870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sc2p8" event={"ID":"ec0ea45e-427b-42b5-a095-2d6e8a0ef43f","Type":"ContainerStarted","Data":"67b67fa468d2d6db9f310f26d67fe01a7dc4de7de148db714d43ce5a73b47d63"} Jul 16 17:32:59.855327 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.855307 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f1735a04c5831a09a654d84791ec904" containerID="1a0a9fffca9bfd17e3922c4542e65f36f44f0e2d48c51f5d8a76cab5c629b0c4" exitCode=0 Jul 16 17:32:59.855413 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.855383 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" event={"ID":"0f1735a04c5831a09a654d84791ec904","Type":"ContainerDied","Data":"1a0a9fffca9bfd17e3922c4542e65f36f44f0e2d48c51f5d8a76cab5c629b0c4"} Jul 16 17:32:59.856896 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.856875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n2xdq" event={"ID":"0e4227b6-a431-4403-98c3-f70f2a1d0c7a","Type":"ContainerStarted","Data":"b02da557a6719b1a054d83092b94511de618782f08357d9151272cc5c61eca15"} Jul 16 17:32:59.868419 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.868373 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.ec2.internal" podStartSLOduration=22.868358505 podStartE2EDuration="22.868358505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:40.823451358 +0000 UTC m=+4.485014992" watchObservedRunningTime="2026-07-16 17:32:59.868358505 +0000 UTC m=+23.529922187" Jul 16 17:32:59.882676 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.882635 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qzrj4" podStartSLOduration=3.068230248 podStartE2EDuration="22.882619864s" totalImagesPullingTime="19.814389616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:59.882115666 +0000 UTC m=+23.543679299" watchObservedRunningTime="2026-07-16 17:32:59.882619864 +0000 UTC m=+23.544183497" Jul 16 17:32:59.901319 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.901275 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sc2p8" podStartSLOduration=4.043314619 podStartE2EDuration="23.901261486s" totalImagesPullingTime="19.857946867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:59.901141014 +0000 UTC m=+23.562704647" watchObservedRunningTime="2026-07-16 17:32:59.901261486 +0000 UTC m=+23.562825119" Jul 16 17:32:59.918555 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.918506 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5z4tz" podStartSLOduration=4.074077817 podStartE2EDuration="23.918485965s" totalImagesPullingTime="19.844408148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:59.918124156 +0000 UTC m=+23.579687789" watchObservedRunningTime="2026-07-16 17:32:59.918485965 +0000 UTC m=+23.580049600" Jul 16 17:32:59.958982 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.958939 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dfr8c" podStartSLOduration=3.127887251 podStartE2EDuration="22.958923381s" totalImagesPullingTime="19.83103613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:59.936583256 +0000 UTC m=+23.598146890" watchObservedRunningTime="2026-07-16 17:32:59.958923381 +0000 UTC m=+23.620487014" Jul 16 17:32:59.981449 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:32:59.981400 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n2xdq" podStartSLOduration=3.125115757 podStartE2EDuration="22.981383773s" totalImagesPullingTime="19.856268016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:59.981217366 +0000 UTC m=+23.642781023" watchObservedRunningTime="2026-07-16 17:32:59.981383773 +0000 UTC m=+23.642947409" Jul 16 17:33:00.766382 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.766349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:00.766601 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.766350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:00.766601 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:00.766486 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:33:00.766601 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.766351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:00.766601 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:00.766567 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:33:00.766842 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:00.766670 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:33:00.863419 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.863390 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:33:00.863905 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.863803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerStarted","Data":"f05f73f95744444899dc7b6bf9431e4e0844f5cbbed6885cc5e05c255d7260c4"} Jul 16 17:33:00.863905 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.863867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerStarted","Data":"255709c0d02fb7e48587cd41728c37a84114006f7963b493e7eb246d11d1a1fc"} Jul 16 17:33:00.863988 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.863918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerStarted","Data":"03af82a966fe45d2e7a0b46b9b195fd6d4a06753983f01e1c3760a272d613d9f"} Jul 16 17:33:00.865951 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.865920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" event={"ID":"0f1735a04c5831a09a654d84791ec904","Type":"ContainerStarted","Data":"2aed40188a2b3d6adfe0fe0d4676e4807fe740f2404d81f370edd04acb176d13"} Jul 16 17:33:00.867531 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.867502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lzhl7" event={"ID":"7906aa23-cf11-4360-b0d0-a2e3a580bffa","Type":"ContainerStarted","Data":"f3068b464f03063b3d7c1e298372a8aced5febbe5b153f16551597d528404533"} Jul 16 17:33:00.885797 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.885737 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.ec2.internal" podStartSLOduration=23.885717228 podStartE2EDuration="23.885717228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:00.885228791 +0000 UTC m=+24.546792429" watchObservedRunningTime="2026-07-16 17:33:00.885717228 +0000 UTC m=+24.547280853" Jul 16 17:33:00.900093 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:00.900046 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lzhl7" podStartSLOduration=5.053322685 podStartE2EDuration="24.900034336s" totalImagesPullingTime="19.846711651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:00.899910007 +0000 UTC m=+24.561473640" watchObservedRunningTime="2026-07-16 17:33:00.900034336 +0000 UTC m=+24.561597968" Jul 16 17:33:01.089568 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:01.089309 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 17:33:01.706257 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:01.706153 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:33:01.089330372Z","UUID":"4a38a70f-5f0f-49c8-b393-da2dc4ca8943","Handler":null,"Name":"","Endpoint":""} Jul 16 17:33:01.707731 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:01.707689 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 17:33:01.707851 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:01.707745 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 17:33:01.873870 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:01.873819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" event={"ID":"df3b7576-550d-40c1-b516-c449e21acaca","Type":"ContainerStarted","Data":"96008bdab29f0f0f7cf41599d4f5c89661be976335d54ac2598e16f01e8d2f7c"} Jul 16 17:33:02.765849 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:02.765628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:02.766031 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:02.765635 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:02.766031 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:02.765635 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:02.766031 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:02.765959 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:33:02.766193 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:02.766087 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:33:02.766237 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:02.766202 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:33:02.879415 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:02.879388 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:33:02.879866 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:02.879837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerStarted","Data":"834531ea684fc2ee8313157f08428f1a9385f322422749bc5b26ee57ea3b9a3b"} Jul 16 17:33:03.287190 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:03.287151 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:33:03.287898 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:03.287870 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:33:03.882458 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:03.882429 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:33:03.882983 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:03.882960 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qzrj4" Jul 16 17:33:04.765844 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.765526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:04.765844 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.765576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:04.766040 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:04.765849 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:33:04.766040 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.765527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:04.766040 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:04.765914 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:33:04.766040 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:04.765989 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:33:04.887412 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.887383 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:33:04.888280 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.887729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerStarted","Data":"7da2e42c91f226cc7ddd9b727a4f3ac600b527f2f6c10a01f8830e3b0ad72729"} Jul 16 17:33:04.888280 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.887959 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:33:04.888280 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.888026 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:33:04.888280 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.888075 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:33:04.888280 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.888169 2576 scope.go:117] "RemoveContainer" containerID="ec2c89ac6d811936f6203b2c4e6e6c8ecfe1ea5119d68616b5cc52442e337c7f" Jul 16 17:33:04.890181 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.890150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" event={"ID":"df3b7576-550d-40c1-b516-c449e21acaca","Type":"ContainerStarted","Data":"0f7d0dbaf71ae7c73426bf1595d513d542cf91b5a0eebc91e9bc782c65e83025"} Jul 16 17:33:04.892221 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.892196 2576 generic.go:356] "Generic (PLEG): container finished" podID="3463dcd4-e95b-46ee-b1a0-a1c242cc91ef" containerID="93e15cd23a7419dced5b5f168e1dc24af45c4ffb0951ef497551e9dea11d188c" exitCode=0 Jul 16 17:33:04.892323 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.892229 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerDied","Data":"93e15cd23a7419dced5b5f168e1dc24af45c4ffb0951ef497551e9dea11d188c"} Jul 16 17:33:04.904964 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.904931 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:33:04.905158 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.905142 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:33:04.994149 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:04.994100 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcztf" podStartSLOduration=3.547878184 podStartE2EDuration="27.994085496s" totalImagesPullingTime="24.446207312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:04.993738375 +0000 UTC m=+28.655302007" watchObservedRunningTime="2026-07-16 17:33:04.994085496 +0000 UTC m=+28.655649129" Jul 16 17:33:05.898239 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:05.898210 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:33:05.899023 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:05.898989 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" event={"ID":"3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607","Type":"ContainerStarted","Data":"c69464caece5e7695b8b949049e559f111684aa6d54eb48daf8e117e3c0d6d93"} Jul 16 17:33:05.936111 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:05.936051 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" podStartSLOduration=8.609827405 podStartE2EDuration="28.936032976s" totalImagesPullingTime="20.326205571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:05.935643535 +0000 UTC m=+29.597207167" watchObservedRunningTime="2026-07-16 17:33:05.936032976 +0000 UTC m=+29.597596608" Jul 16 17:33:06.766206 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.766172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:06.766416 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:06.766257 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:33:06.766588 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.766561 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:06.766731 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.766718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:06.766779 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:06.766724 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:33:06.766819 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:06.766802 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:33:06.796041 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.796008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b5qk9"] Jul 16 17:33:06.799892 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.799851 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65qpf"] Jul 16 17:33:06.800454 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.800415 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dmrj5"] Jul 16 17:33:06.903086 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.903054 2576 generic.go:356] "Generic (PLEG): container finished" podID="3463dcd4-e95b-46ee-b1a0-a1c242cc91ef" containerID="a9c97084e686494726c53d32c803e87466b30221bacc635bba706208d7774561" exitCode=0 Jul 16 17:33:06.903486 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.903143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerDied","Data":"a9c97084e686494726c53d32c803e87466b30221bacc635bba706208d7774561"} Jul 16 17:33:06.903486 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.903178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:06.903486 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.903283 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:06.903486 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:06.903377 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:33:06.903486 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:06.903411 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:33:06.903486 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:06.903437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:06.903774 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:06.903505 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:33:08.765832 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:08.765603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:08.766186 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:08.765603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:08.766186 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:08.765862 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:33:08.766186 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:08.765976 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:33:08.766186 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:08.765681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:08.766186 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:08.766108 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:33:08.909473 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:08.909438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerStarted","Data":"ea53159368723c2f953041556943766d57b623a2fa927071dd1156d218d2ccc0"} Jul 16 17:33:09.913394 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:09.913361 2576 generic.go:356] "Generic (PLEG): container finished" podID="3463dcd4-e95b-46ee-b1a0-a1c242cc91ef" containerID="ea53159368723c2f953041556943766d57b623a2fa927071dd1156d218d2ccc0" exitCode=0 Jul 16 17:33:09.913946 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:09.913398 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerDied","Data":"ea53159368723c2f953041556943766d57b623a2fa927071dd1156d218d2ccc0"} Jul 16 17:33:10.392727 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:10.392686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:10.392881 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.392819 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:10.392881 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.392876 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs podName:8680dbdf-6918-4634-9412-e88acf7f528a nodeName:}" failed. No retries permitted until 2026-07-16 17:33:42.392856562 +0000 UTC m=+66.054420173 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs") pod "network-metrics-daemon-dmrj5" (UID: "8680dbdf-6918-4634-9412-e88acf7f528a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:10.493128 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:10.493091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:10.493329 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.493213 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:10.493329 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.493262 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret podName:cb2e0f7c-1598-4d6f-8515-07473e51b8c9 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:42.493249979 +0000 UTC m=+66.154813591 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret") pod "global-pull-secret-syncer-65qpf" (UID: "cb2e0f7c-1598-4d6f-8515-07473e51b8c9") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:10.594201 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:10.594169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:10.594358 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.594312 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:33:10.594358 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.594332 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:33:10.594358 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.594345 2576 projected.go:196] Error preparing data for projected volume kube-api-access-rjrsw for pod openshift-network-diagnostics/network-check-target-b5qk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:10.594528 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.594409 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw podName:a4d28a25-bf42-4815-9387-e010faaa08d2 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:42.594387284 +0000 UTC m=+66.255950899 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rjrsw" (UniqueName: "kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw") pod "network-check-target-b5qk9" (UID: "a4d28a25-bf42-4815-9387-e010faaa08d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:10.766055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:10.765940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:10.766055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:10.765967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:10.766055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:10.765968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:10.766291 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.766069 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5qk9" podUID="a4d28a25-bf42-4815-9387-e010faaa08d2" Jul 16 17:33:10.766291 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.766186 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmrj5" podUID="8680dbdf-6918-4634-9412-e88acf7f528a" Jul 16 17:33:10.766291 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:10.766283 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qpf" podUID="cb2e0f7c-1598-4d6f-8515-07473e51b8c9" Jul 16 17:33:11.712322 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.712292 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-217.ec2.internal" event="NodeReady" Jul 16 17:33:11.712832 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.712440 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 17:33:11.755179 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.755142 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-846b6cfbb5-9jhz2"] Jul 16 17:33:11.758842 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.758813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:11.761376 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.761354 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 17:33:11.761503 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.761404 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 17:33:11.761503 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.761466 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 17:33:11.761627 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.761520 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dx8hw\"" Jul 16 17:33:11.767450 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.767427 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 17:33:11.775260 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.775239 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xtq5p"] Jul 16 17:33:11.779226 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.779193 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cbc5s"] Jul 16 17:33:11.779384 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.779349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:11.781901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.781878 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7tn9x\"" Jul 16 17:33:11.782287 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.782271 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 17:33:11.782390 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.782312 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 17:33:11.782934 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.782916 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-846b6cfbb5-9jhz2"] Jul 16 17:33:11.783056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.783019 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cbc5s" Jul 16 17:33:11.785139 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.785063 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 17:33:11.785251 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.785177 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 17:33:11.785311 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.785294 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4ccwt\"" Jul 16 17:33:11.785379 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.785360 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 17:33:11.791951 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.791887 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cbc5s"] Jul 16 17:33:11.792281 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.792260 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xtq5p"] Jul 16 17:33:11.905129 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6jp\" (UniqueName: \"kubernetes.io/projected/86b5d1c9-6f4b-4898-8758-b354d8af0c38-kube-api-access-4d6jp\") pod \"ingress-canary-cbc5s\" (UID: \"86b5d1c9-6f4b-4898-8758-b354d8af0c38\") " pod="openshift-ingress-canary/ingress-canary-cbc5s" Jul 16 17:33:11.905331 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-installation-pull-secrets\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:11.905331 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-image-registry-private-configuration\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:11.905331 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-bound-sa-token\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:11.905331 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86b5d1c9-6f4b-4898-8758-b354d8af0c38-cert\") pod \"ingress-canary-cbc5s\" (UID: \"86b5d1c9-6f4b-4898-8758-b354d8af0c38\") " pod="openshift-ingress-canary/ingress-canary-cbc5s" Jul 16 17:33:11.905331 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-tmp-dir\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:11.905589 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rclqz\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-kube-api-access-rclqz\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:11.905589 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:11.905589 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-certificates\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:11.905589 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-config-volume\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:11.905589 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905581 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a8a4c22-ae74-44b4-87bb-807398b17251-ca-trust-extracted\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:11.905820 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-trusted-ca\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:11.905820 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:11.905820 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:11.905663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlx5s\" (UniqueName: \"kubernetes.io/projected/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-kube-api-access-xlx5s\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:12.006446 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-tmp-dir\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:12.006631 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rclqz\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-kube-api-access-rclqz\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.006631 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.006631 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-certificates\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.006631 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-config-volume\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:12.006631 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a8a4c22-ae74-44b4-87bb-807398b17251-ca-trust-extracted\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.006631 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-trusted-ca\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.006643 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.006665 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846b6cfbb5-9jhz2: secret "image-registry-tls" not found Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.006744 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls podName:4a8a4c22-ae74-44b4-87bb-807398b17251 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:12.506720782 +0000 UTC m=+36.168284409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls") pod "image-registry-846b6cfbb5-9jhz2" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251") : secret "image-registry-tls" not found Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.006743 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.006782 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls podName:8ce9162f-6d4d-44ec-a1ea-4d5e2652c758 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:12.506772388 +0000 UTC m=+36.168336015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls") pod "dns-default-xtq5p" (UID: "8ce9162f-6d4d-44ec-a1ea-4d5e2652c758") : secret "dns-default-metrics-tls" not found Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlx5s\" (UniqueName: \"kubernetes.io/projected/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-kube-api-access-xlx5s\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6jp\" (UniqueName: \"kubernetes.io/projected/86b5d1c9-6f4b-4898-8758-b354d8af0c38-kube-api-access-4d6jp\") pod \"ingress-canary-cbc5s\" (UID: \"86b5d1c9-6f4b-4898-8758-b354d8af0c38\") " pod="openshift-ingress-canary/ingress-canary-cbc5s" Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-installation-pull-secrets\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.006957 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-image-registry-private-configuration\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.007420 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-bound-sa-token\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.007420 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86b5d1c9-6f4b-4898-8758-b354d8af0c38-cert\") pod \"ingress-canary-cbc5s\" (UID: \"86b5d1c9-6f4b-4898-8758-b354d8af0c38\") " pod="openshift-ingress-canary/ingress-canary-cbc5s" Jul 16 17:33:12.007420 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.006996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-tmp-dir\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:12.007420 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.007250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-certificates\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.007616 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.007570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-config-volume\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:12.007960 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.007814 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a8a4c22-ae74-44b4-87bb-807398b17251-ca-trust-extracted\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.008286 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.008110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-trusted-ca\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.011339 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.011320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-installation-pull-secrets\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.011339 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.011333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-image-registry-private-configuration\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.011489 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.011455 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86b5d1c9-6f4b-4898-8758-b354d8af0c38-cert\") pod \"ingress-canary-cbc5s\" (UID: \"86b5d1c9-6f4b-4898-8758-b354d8af0c38\") " pod="openshift-ingress-canary/ingress-canary-cbc5s" Jul 16 17:33:12.023347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.021124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlx5s\" (UniqueName: \"kubernetes.io/projected/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-kube-api-access-xlx5s\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:12.023347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.021243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-bound-sa-token\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.023347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.021338 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6jp\" (UniqueName: \"kubernetes.io/projected/86b5d1c9-6f4b-4898-8758-b354d8af0c38-kube-api-access-4d6jp\") pod \"ingress-canary-cbc5s\" (UID: \"86b5d1c9-6f4b-4898-8758-b354d8af0c38\") " pod="openshift-ingress-canary/ingress-canary-cbc5s" Jul 16 17:33:12.023347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.021468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rclqz\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-kube-api-access-rclqz\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.100180 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.100135 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cbc5s" Jul 16 17:33:12.261539 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.261371 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cbc5s"] Jul 16 17:33:12.512734 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.512592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:12.512734 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.512721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:12.512967 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.512814 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:33:12.512967 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.512832 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:33:12.512967 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.512838 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846b6cfbb5-9jhz2: secret "image-registry-tls" not found Jul 16 17:33:12.512967 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.512899 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls podName:8ce9162f-6d4d-44ec-a1ea-4d5e2652c758 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:13.512883074 +0000 UTC m=+37.174446684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls") pod "dns-default-xtq5p" (UID: "8ce9162f-6d4d-44ec-a1ea-4d5e2652c758") : secret "dns-default-metrics-tls" not found Jul 16 17:33:12.512967 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:12.512915 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls podName:4a8a4c22-ae74-44b4-87bb-807398b17251 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:13.512908919 +0000 UTC m=+37.174472530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls") pod "image-registry-846b6cfbb5-9jhz2" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251") : secret "image-registry-tls" not found Jul 16 17:33:12.766460 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.766372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:12.766460 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.766413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:12.767144 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.766372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:12.770745 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.770482 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:33:12.770745 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.770516 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:33:12.770745 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.770516 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-455gg\"" Jul 16 17:33:12.770745 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.770602 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:33:12.770745 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.770613 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:33:12.771068 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:12.770867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ck2g5\"" Jul 16 17:33:13.538257 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:13.538218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:13.538459 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:13.538291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:13.538459 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:13.538388 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:33:13.538459 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:13.538401 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:33:13.538459 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:13.538415 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846b6cfbb5-9jhz2: secret "image-registry-tls" not found Jul 16 17:33:13.538660 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:13.538464 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls podName:4a8a4c22-ae74-44b4-87bb-807398b17251 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:15.538445449 +0000 UTC m=+39.200009080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls") pod "image-registry-846b6cfbb5-9jhz2" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251") : secret "image-registry-tls" not found Jul 16 17:33:13.538660 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:13.538481 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls podName:8ce9162f-6d4d-44ec-a1ea-4d5e2652c758 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:15.538472557 +0000 UTC m=+39.200036171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls") pod "dns-default-xtq5p" (UID: "8ce9162f-6d4d-44ec-a1ea-4d5e2652c758") : secret "dns-default-metrics-tls" not found Jul 16 17:33:13.746502 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:33:13.746473 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b5d1c9_6f4b_4898_8758_b354d8af0c38.slice/crio-660e088f48a7cb4318952194def43bf8524780c9d333a59a7515e41304378d46 WatchSource:0}: Error finding container 660e088f48a7cb4318952194def43bf8524780c9d333a59a7515e41304378d46: Status 404 returned error can't find the container with id 660e088f48a7cb4318952194def43bf8524780c9d333a59a7515e41304378d46 Jul 16 17:33:13.926911 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:13.926876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerStarted","Data":"35a031711a65804fe03a6b90a04d4624783b56a6c9af605fd02fd154e99a659c"} Jul 16 17:33:13.928043 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:13.928021 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cbc5s" event={"ID":"86b5d1c9-6f4b-4898-8758-b354d8af0c38","Type":"ContainerStarted","Data":"660e088f48a7cb4318952194def43bf8524780c9d333a59a7515e41304378d46"} Jul 16 17:33:14.933850 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:14.933816 2576 generic.go:356] "Generic (PLEG): container finished" podID="3463dcd4-e95b-46ee-b1a0-a1c242cc91ef" containerID="35a031711a65804fe03a6b90a04d4624783b56a6c9af605fd02fd154e99a659c" exitCode=0 Jul 16 17:33:14.934317 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:14.933879 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerDied","Data":"35a031711a65804fe03a6b90a04d4624783b56a6c9af605fd02fd154e99a659c"} Jul 16 17:33:15.553563 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:15.553521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:15.553758 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:15.553617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:15.553758 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:15.553661 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:33:15.553758 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:15.553749 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls podName:8ce9162f-6d4d-44ec-a1ea-4d5e2652c758 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:19.553731291 +0000 UTC m=+43.215294903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls") pod "dns-default-xtq5p" (UID: "8ce9162f-6d4d-44ec-a1ea-4d5e2652c758") : secret "dns-default-metrics-tls" not found Jul 16 17:33:15.553758 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:15.553748 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:33:15.553934 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:15.553766 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846b6cfbb5-9jhz2: secret "image-registry-tls" not found Jul 16 17:33:15.553934 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:15.553808 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls podName:4a8a4c22-ae74-44b4-87bb-807398b17251 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:19.553796532 +0000 UTC m=+43.215360166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls") pod "image-registry-846b6cfbb5-9jhz2" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251") : secret "image-registry-tls" not found Jul 16 17:33:15.939872 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:15.939826 2576 generic.go:356] "Generic (PLEG): container finished" podID="3463dcd4-e95b-46ee-b1a0-a1c242cc91ef" containerID="9dfe1b9a1a4f1acb67b5fae68f60f76be17085513386efc9ad631590ab1ab303" exitCode=0 Jul 16 17:33:15.940271 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:15.939884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerDied","Data":"9dfe1b9a1a4f1acb67b5fae68f60f76be17085513386efc9ad631590ab1ab303"} Jul 16 17:33:16.945524 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:16.945343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-czpbw" event={"ID":"3463dcd4-e95b-46ee-b1a0-a1c242cc91ef","Type":"ContainerStarted","Data":"976c61a6af6c0b5fb57841a758731b38e2fbdf88487117fd3eac342c4838aad5"} Jul 16 17:33:16.981018 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:16.980964 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-czpbw" podStartSLOduration=9.723585218 podStartE2EDuration="40.980946759s" totalImagesPullingTime="31.257361541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:16.979604135 +0000 UTC m=+40.641167767" watchObservedRunningTime="2026-07-16 17:33:16.980946759 +0000 UTC m=+40.642510392" Jul 16 17:33:17.948885 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:17.948848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cbc5s" event={"ID":"86b5d1c9-6f4b-4898-8758-b354d8af0c38","Type":"ContainerStarted","Data":"32337f0b6f87061afc8b898ff7e55d6f2a46baa83719afcac4edf993987ad25a"} Jul 16 17:33:17.981165 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:17.981110 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cbc5s" podStartSLOduration=3.745167028 podStartE2EDuration="6.981094203s" totalImagesPullingTime="3.235927175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:17.980948674 +0000 UTC m=+41.642512307" watchObservedRunningTime="2026-07-16 17:33:17.981094203 +0000 UTC m=+41.642657835" Jul 16 17:33:19.582265 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:19.582217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:19.582644 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:19.582293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:19.582644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:19.582368 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:33:19.582644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:19.582388 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846b6cfbb5-9jhz2: secret "image-registry-tls" not found Jul 16 17:33:19.582644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:19.582412 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:33:19.582644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:19.582442 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls podName:4a8a4c22-ae74-44b4-87bb-807398b17251 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:27.582427011 +0000 UTC m=+51.243990625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls") pod "image-registry-846b6cfbb5-9jhz2" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251") : secret "image-registry-tls" not found Jul 16 17:33:19.582644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:19.582456 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls podName:8ce9162f-6d4d-44ec-a1ea-4d5e2652c758 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:27.58245032 +0000 UTC m=+51.244013931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls") pod "dns-default-xtq5p" (UID: "8ce9162f-6d4d-44ec-a1ea-4d5e2652c758") : secret "dns-default-metrics-tls" not found Jul 16 17:33:27.637179 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:27.637134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:27.637604 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:27.637224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:27.637604 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:27.637321 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:33:27.637604 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:27.637344 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846b6cfbb5-9jhz2: secret "image-registry-tls" not found Jul 16 17:33:27.637604 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:27.637387 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:33:27.637604 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:27.637458 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls podName:4a8a4c22-ae74-44b4-87bb-807398b17251 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:43.637441856 +0000 UTC m=+67.299005467 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls") pod "image-registry-846b6cfbb5-9jhz2" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251") : secret "image-registry-tls" not found Jul 16 17:33:27.637604 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:27.637472 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls podName:8ce9162f-6d4d-44ec-a1ea-4d5e2652c758 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:43.637466621 +0000 UTC m=+67.299030231 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls") pod "dns-default-xtq5p" (UID: "8ce9162f-6d4d-44ec-a1ea-4d5e2652c758") : secret "dns-default-metrics-tls" not found Jul 16 17:33:36.913762 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:36.913726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8ksg" Jul 16 17:33:42.441784 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.441744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:33:42.445876 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.445859 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:33:42.452418 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:42.452400 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 17:33:42.452490 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:42.452460 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs podName:8680dbdf-6918-4634-9412-e88acf7f528a nodeName:}" failed. No retries permitted until 2026-07-16 17:34:46.452441158 +0000 UTC m=+130.114004769 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs") pod "network-metrics-daemon-dmrj5" (UID: "8680dbdf-6918-4634-9412-e88acf7f528a") : secret "metrics-daemon-secret" not found Jul 16 17:33:42.542571 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.542544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:42.546163 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.546142 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:33:42.556912 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.556887 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb2e0f7c-1598-4d6f-8515-07473e51b8c9-original-pull-secret\") pod \"global-pull-secret-syncer-65qpf\" (UID: \"cb2e0f7c-1598-4d6f-8515-07473e51b8c9\") " pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:42.643600 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.643561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:42.646528 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.646510 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:33:42.656865 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.656842 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:33:42.667035 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.667010 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrsw\" (UniqueName: \"kubernetes.io/projected/a4d28a25-bf42-4815-9387-e010faaa08d2-kube-api-access-rjrsw\") pod \"network-check-target-b5qk9\" (UID: \"a4d28a25-bf42-4815-9387-e010faaa08d2\") " pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:42.778582 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.778504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qpf" Jul 16 17:33:42.794427 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.794399 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ck2g5\"" Jul 16 17:33:42.802003 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.801968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:42.914605 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.914558 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65qpf"] Jul 16 17:33:42.917651 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:33:42.917624 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2e0f7c_1598_4d6f_8515_07473e51b8c9.slice/crio-a69f935a70ddf38ad14492c44ea32f8b0a952bd0391f8e55aff0526fe23e3589 WatchSource:0}: Error finding container a69f935a70ddf38ad14492c44ea32f8b0a952bd0391f8e55aff0526fe23e3589: Status 404 returned error can't find the container with id a69f935a70ddf38ad14492c44ea32f8b0a952bd0391f8e55aff0526fe23e3589 Jul 16 17:33:42.957418 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:42.957376 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b5qk9"] Jul 16 17:33:42.961840 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:33:42.961814 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d28a25_bf42_4815_9387_e010faaa08d2.slice/crio-d4c1ff770a12ada8072de3b65e6b5643f88fee6ca9cc2f4b2afc0f2b78d5cd3f WatchSource:0}: Error finding container d4c1ff770a12ada8072de3b65e6b5643f88fee6ca9cc2f4b2afc0f2b78d5cd3f: Status 404 returned error can't find the container with id d4c1ff770a12ada8072de3b65e6b5643f88fee6ca9cc2f4b2afc0f2b78d5cd3f Jul 16 17:33:43.007692 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:43.007656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b5qk9" event={"ID":"a4d28a25-bf42-4815-9387-e010faaa08d2","Type":"ContainerStarted","Data":"d4c1ff770a12ada8072de3b65e6b5643f88fee6ca9cc2f4b2afc0f2b78d5cd3f"} Jul 16 17:33:43.008648 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:43.008624 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-65qpf" event={"ID":"cb2e0f7c-1598-4d6f-8515-07473e51b8c9","Type":"ContainerStarted","Data":"a69f935a70ddf38ad14492c44ea32f8b0a952bd0391f8e55aff0526fe23e3589"} Jul 16 17:33:43.650231 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:43.650187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:33:43.650644 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:43.650270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:33:43.650644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:43.650383 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:33:43.650644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:43.650407 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846b6cfbb5-9jhz2: secret "image-registry-tls" not found Jul 16 17:33:43.650644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:43.650410 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:33:43.650644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:43.650477 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls podName:8ce9162f-6d4d-44ec-a1ea-4d5e2652c758 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:15.650455088 +0000 UTC m=+99.312018712 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls") pod "dns-default-xtq5p" (UID: "8ce9162f-6d4d-44ec-a1ea-4d5e2652c758") : secret "dns-default-metrics-tls" not found Jul 16 17:33:43.650644 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:33:43.650497 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls podName:4a8a4c22-ae74-44b4-87bb-807398b17251 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:15.650488954 +0000 UTC m=+99.312052574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls") pod "image-registry-846b6cfbb5-9jhz2" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251") : secret "image-registry-tls" not found Jul 16 17:33:48.027654 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:48.027614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b5qk9" event={"ID":"a4d28a25-bf42-4815-9387-e010faaa08d2","Type":"ContainerStarted","Data":"a825b8ab87de9edddee158e27d977acbbfdff0c7a836e00bc715f014db0a219e"} Jul 16 17:33:48.028117 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:48.027719 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:33:48.029217 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:48.029194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-65qpf" event={"ID":"cb2e0f7c-1598-4d6f-8515-07473e51b8c9","Type":"ContainerStarted","Data":"7d81c4cfb551f9342a4bbd661e65d4211530058594723d633a1f6c70e216c242"} Jul 16 17:33:48.050128 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:48.050027 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b5qk9" podStartSLOduration=66.228035453 podStartE2EDuration="1m11.050012332s" totalImagesPullingTime="4.821976879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:48.048558655 +0000 UTC m=+71.710122269" watchObservedRunningTime="2026-07-16 17:33:48.050012332 +0000 UTC m=+71.711575943" Jul 16 17:33:48.068064 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:33:48.068010 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-65qpf" podStartSLOduration=66.190768983 podStartE2EDuration="1m11.067993472s" totalImagesPullingTime="4.877224489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:48.067388683 +0000 UTC m=+71.728952322" watchObservedRunningTime="2026-07-16 17:33:48.067993472 +0000 UTC m=+71.729557129" Jul 16 17:34:13.540901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.540780 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl"] Jul 16 17:34:13.545005 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.544986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.549270 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.549245 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 16 17:34:13.549405 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.549305 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 16 17:34:13.550526 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.550505 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 16 17:34:13.550747 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.550511 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:34:13.551343 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.551321 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gwnnx\"" Jul 16 17:34:13.602913 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.602879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl"] Jul 16 17:34:13.624879 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.624841 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4"] Jul 16 17:34:13.629027 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.628997 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4" Jul 16 17:34:13.637416 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.637392 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pss8q\"" Jul 16 17:34:13.642942 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.642920 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4"] Jul 16 17:34:13.667556 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.667519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzhgf\" (UniqueName: \"kubernetes.io/projected/30766de9-6ead-4f64-a8ba-5018b71317c1-kube-api-access-dzhgf\") pod \"network-check-source-869dcfcdcd-2z7l4\" (UID: \"30766de9-6ead-4f64-a8ba-5018b71317c1\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4" Jul 16 17:34:13.667739 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.667589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6feb2440-11a1-4c36-9d88-bcd0e715062a-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-x48rl\" (UID: \"6feb2440-11a1-4c36-9d88-bcd0e715062a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.667739 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.667607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7558\" (UniqueName: \"kubernetes.io/projected/6feb2440-11a1-4c36-9d88-bcd0e715062a-kube-api-access-p7558\") pod \"kube-storage-version-migrator-operator-799cb7f958-x48rl\" (UID: \"6feb2440-11a1-4c36-9d88-bcd0e715062a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.667739 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.667629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6feb2440-11a1-4c36-9d88-bcd0e715062a-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-x48rl\" (UID: \"6feb2440-11a1-4c36-9d88-bcd0e715062a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.767973 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.767930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6feb2440-11a1-4c36-9d88-bcd0e715062a-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-x48rl\" (UID: \"6feb2440-11a1-4c36-9d88-bcd0e715062a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.767973 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.767969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7558\" (UniqueName: \"kubernetes.io/projected/6feb2440-11a1-4c36-9d88-bcd0e715062a-kube-api-access-p7558\") pod \"kube-storage-version-migrator-operator-799cb7f958-x48rl\" (UID: \"6feb2440-11a1-4c36-9d88-bcd0e715062a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.768177 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.767997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6feb2440-11a1-4c36-9d88-bcd0e715062a-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-x48rl\" (UID: \"6feb2440-11a1-4c36-9d88-bcd0e715062a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.768212 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.768188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzhgf\" (UniqueName: \"kubernetes.io/projected/30766de9-6ead-4f64-a8ba-5018b71317c1-kube-api-access-dzhgf\") pod \"network-check-source-869dcfcdcd-2z7l4\" (UID: \"30766de9-6ead-4f64-a8ba-5018b71317c1\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4" Jul 16 17:34:13.768621 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.768593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6feb2440-11a1-4c36-9d88-bcd0e715062a-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-x48rl\" (UID: \"6feb2440-11a1-4c36-9d88-bcd0e715062a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.770221 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.770187 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6feb2440-11a1-4c36-9d88-bcd0e715062a-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-x48rl\" (UID: \"6feb2440-11a1-4c36-9d88-bcd0e715062a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.782916 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.782881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7558\" (UniqueName: \"kubernetes.io/projected/6feb2440-11a1-4c36-9d88-bcd0e715062a-kube-api-access-p7558\") pod \"kube-storage-version-migrator-operator-799cb7f958-x48rl\" (UID: \"6feb2440-11a1-4c36-9d88-bcd0e715062a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.783042 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.782923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzhgf\" (UniqueName: \"kubernetes.io/projected/30766de9-6ead-4f64-a8ba-5018b71317c1-kube-api-access-dzhgf\") pod \"network-check-source-869dcfcdcd-2z7l4\" (UID: \"30766de9-6ead-4f64-a8ba-5018b71317c1\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4" Jul 16 17:34:13.853586 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.853504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" Jul 16 17:34:13.936959 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.936925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4" Jul 16 17:34:13.989422 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:13.989112 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl"] Jul 16 17:34:13.993558 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:34:13.993513 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6feb2440_11a1_4c36_9d88_bcd0e715062a.slice/crio-567b186b0e2787dfefb680e4faf5354fd39e10de43ca2ef82bd032f040b15ec0 WatchSource:0}: Error finding container 567b186b0e2787dfefb680e4faf5354fd39e10de43ca2ef82bd032f040b15ec0: Status 404 returned error can't find the container with id 567b186b0e2787dfefb680e4faf5354fd39e10de43ca2ef82bd032f040b15ec0 Jul 16 17:34:14.067579 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:14.067544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4"] Jul 16 17:34:14.070340 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:34:14.070311 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30766de9_6ead_4f64_a8ba_5018b71317c1.slice/crio-9d37996da718610a999565829988a2f24926b0f59c16df2af0eabda2571445d7 WatchSource:0}: Error finding container 9d37996da718610a999565829988a2f24926b0f59c16df2af0eabda2571445d7: Status 404 returned error can't find the container with id 9d37996da718610a999565829988a2f24926b0f59c16df2af0eabda2571445d7 Jul 16 17:34:14.090541 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:14.090504 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" event={"ID":"6feb2440-11a1-4c36-9d88-bcd0e715062a","Type":"ContainerStarted","Data":"567b186b0e2787dfefb680e4faf5354fd39e10de43ca2ef82bd032f040b15ec0"} Jul 16 17:34:14.091429 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:14.091409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4" event={"ID":"30766de9-6ead-4f64-a8ba-5018b71317c1","Type":"ContainerStarted","Data":"9d37996da718610a999565829988a2f24926b0f59c16df2af0eabda2571445d7"} Jul 16 17:34:15.097571 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:15.097502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4" event={"ID":"30766de9-6ead-4f64-a8ba-5018b71317c1","Type":"ContainerStarted","Data":"fcd3f9cfb7de9873fb04e8c1ad43cfb1cf13bee91328696f41eb60797d394474"} Jul 16 17:34:15.114986 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:15.114925 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2z7l4" podStartSLOduration=2.114903487 podStartE2EDuration="2.114903487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:15.114358966 +0000 UTC m=+98.775922598" watchObservedRunningTime="2026-07-16 17:34:15.114903487 +0000 UTC m=+98.776467120" Jul 16 17:34:15.685057 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:15.685016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:34:15.685260 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:15.685080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") pod \"image-registry-846b6cfbb5-9jhz2\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:34:15.685260 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:15.685175 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:34:15.685260 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:15.685176 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:15.685421 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:15.685262 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls podName:8ce9162f-6d4d-44ec-a1ea-4d5e2652c758 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.685243253 +0000 UTC m=+163.346806872 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls") pod "dns-default-xtq5p" (UID: "8ce9162f-6d4d-44ec-a1ea-4d5e2652c758") : secret "dns-default-metrics-tls" not found Jul 16 17:34:15.685421 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:15.685187 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846b6cfbb5-9jhz2: secret "image-registry-tls" not found Jul 16 17:34:15.685421 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:15.685307 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls podName:4a8a4c22-ae74-44b4-87bb-807398b17251 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.685296328 +0000 UTC m=+163.346859939 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls") pod "image-registry-846b6cfbb5-9jhz2" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251") : secret "image-registry-tls" not found Jul 16 17:34:17.105962 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:17.105924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" event={"ID":"6feb2440-11a1-4c36-9d88-bcd0e715062a","Type":"ContainerStarted","Data":"fa351f78287fb2c6a1acca50b57636c099eb3355ad998c6b096161765823e667"} Jul 16 17:34:17.129824 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:17.129774 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" podStartSLOduration=1.891883141 podStartE2EDuration="4.12975814s" totalImagesPullingTime="2.237874999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:17.129216217 +0000 UTC m=+100.790779851" watchObservedRunningTime="2026-07-16 17:34:17.12975814 +0000 UTC m=+100.791321774" Jul 16 17:34:18.651119 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.651084 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-spllq"] Jul 16 17:34:18.654205 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.654188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.657038 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.657011 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 17:34:18.657038 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.657022 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 17:34:18.657921 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.657899 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 17:34:18.657921 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.657910 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 17:34:18.658088 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.657901 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxz8n\"" Jul 16 17:34:18.674513 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.674484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-spllq"] Jul 16 17:34:18.707329 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.707292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd09d19f-aca3-43a5-ba18-82f2b996dba4-crio-socket\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.707329 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.707334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd09d19f-aca3-43a5-ba18-82f2b996dba4-data-volume\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.707550 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.707427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.707550 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.707462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.707550 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.707492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4dq\" (UniqueName: \"kubernetes.io/projected/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-api-access-7l4dq\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.808047 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.808003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd09d19f-aca3-43a5-ba18-82f2b996dba4-crio-socket\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.808047 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.808046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd09d19f-aca3-43a5-ba18-82f2b996dba4-data-volume\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.808277 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.808105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.808277 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.808126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.808277 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.808136 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd09d19f-aca3-43a5-ba18-82f2b996dba4-crio-socket\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.808277 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.808148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4dq\" (UniqueName: \"kubernetes.io/projected/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-api-access-7l4dq\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.808277 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:18.808230 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:34:18.808277 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:18.808246 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:34:18.808504 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:18.808307 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:19.308287004 +0000 UTC m=+102.969850619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : configmap "kube-rbac-proxy" not found Jul 16 17:34:18.808504 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:18.808324 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:19.308316234 +0000 UTC m=+102.969879848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : secret "insights-runtime-extractor-tls" not found Jul 16 17:34:18.808504 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.808443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd09d19f-aca3-43a5-ba18-82f2b996dba4-data-volume\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:18.821218 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:18.821192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4dq\" (UniqueName: \"kubernetes.io/projected/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-api-access-7l4dq\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:19.033908 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:19.033834 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b5qk9" Jul 16 17:34:19.312173 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:19.312079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:19.312173 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:19.312123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:19.312424 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:19.312209 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:34:19.312424 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:19.312233 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:34:19.312424 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:19.312276 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:20.312257867 +0000 UTC m=+103.973821500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : configmap "kube-rbac-proxy" not found Jul 16 17:34:19.312424 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:19.312295 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:20.312285753 +0000 UTC m=+103.973849364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : secret "insights-runtime-extractor-tls" not found Jul 16 17:34:20.319948 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:20.319914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:20.319948 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:20.319955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:20.320502 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:20.320474 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:34:20.320612 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:20.320599 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:22.320565652 +0000 UTC m=+105.982129268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : secret "insights-runtime-extractor-tls" not found Jul 16 17:34:20.321066 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:20.320473 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:34:20.321158 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:20.321107 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:22.32108948 +0000 UTC m=+105.982653106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : configmap "kube-rbac-proxy" not found Jul 16 17:34:21.127173 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.127139 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4fxws"] Jul 16 17:34:21.145197 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.145171 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4fxws"] Jul 16 17:34:21.145342 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.145255 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.147951 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.147928 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 16 17:34:21.148161 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.148141 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 16 17:34:21.148294 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.148217 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 16 17:34:21.148294 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.148263 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 16 17:34:21.148760 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.148747 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zcskw\"" Jul 16 17:34:21.227270 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.227232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9-signing-key\") pod \"service-ca-79f68b9f4f-4fxws\" (UID: \"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.227436 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.227298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9-signing-cabundle\") pod \"service-ca-79f68b9f4f-4fxws\" (UID: \"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.227436 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.227348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pk5\" (UniqueName: \"kubernetes.io/projected/d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9-kube-api-access-m4pk5\") pod \"service-ca-79f68b9f4f-4fxws\" (UID: \"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.328617 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.328581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9-signing-cabundle\") pod \"service-ca-79f68b9f4f-4fxws\" (UID: \"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.329049 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.328637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pk5\" (UniqueName: \"kubernetes.io/projected/d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9-kube-api-access-m4pk5\") pod \"service-ca-79f68b9f4f-4fxws\" (UID: \"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.329049 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.328694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9-signing-key\") pod \"service-ca-79f68b9f4f-4fxws\" (UID: \"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.329264 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.329240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9-signing-cabundle\") pod \"service-ca-79f68b9f4f-4fxws\" (UID: \"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.331144 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.331127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9-signing-key\") pod \"service-ca-79f68b9f4f-4fxws\" (UID: \"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.341841 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.341819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pk5\" (UniqueName: \"kubernetes.io/projected/d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9-kube-api-access-m4pk5\") pod \"service-ca-79f68b9f4f-4fxws\" (UID: \"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.426533 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.426509 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dfr8c_4da5bf01-f439-4e51-8ef5-6828fee14726/dns-node-resolver/0.log" Jul 16 17:34:21.453808 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.453775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" Jul 16 17:34:21.595978 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:21.595948 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4fxws"] Jul 16 17:34:21.598853 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:34:21.598830 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ef8a99_71b3_48d7_a866_0a4a6f92dfc9.slice/crio-37870bf9cffbf5b8527d1468d651f96b19441750a345c07623da44cc7bcceec3 WatchSource:0}: Error finding container 37870bf9cffbf5b8527d1468d651f96b19441750a345c07623da44cc7bcceec3: Status 404 returned error can't find the container with id 37870bf9cffbf5b8527d1468d651f96b19441750a345c07623da44cc7bcceec3 Jul 16 17:34:22.122279 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:22.122246 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" event={"ID":"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9","Type":"ContainerStarted","Data":"37870bf9cffbf5b8527d1468d651f96b19441750a345c07623da44cc7bcceec3"} Jul 16 17:34:22.338621 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:22.338581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:22.338621 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:22.338625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:22.339116 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:22.338758 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:34:22.339116 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:22.338798 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:34:22.339116 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:22.338822 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:26.338808256 +0000 UTC m=+110.000371868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : secret "insights-runtime-extractor-tls" not found Jul 16 17:34:22.339116 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:22.338893 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:26.338872115 +0000 UTC m=+110.000435740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : configmap "kube-rbac-proxy" not found Jul 16 17:34:22.624316 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:22.624285 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sc2p8_ec0ea45e-427b-42b5-a095-2d6e8a0ef43f/node-ca/0.log" Jul 16 17:34:23.025190 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:23.025160 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cbc5s_86b5d1c9-6f4b-4898-8758-b354d8af0c38/serve-healthcheck-canary/0.log" Jul 16 17:34:24.027715 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:24.027670 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-x48rl_6feb2440-11a1-4c36-9d88-bcd0e715062a/kube-storage-version-migrator-operator/0.log" Jul 16 17:34:24.130083 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:24.130045 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" event={"ID":"d5ef8a99-71b3-48d7-a866-0a4a6f92dfc9","Type":"ContainerStarted","Data":"30bac1bae5f824ae5ad67df31bc87fecc246525a9947c3d46547d12c304755d7"} Jul 16 17:34:24.149191 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:24.149123 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-4fxws" podStartSLOduration=0.693845803 podStartE2EDuration="3.14910193s" totalImagesPullingTime="2.455256127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:24.148505212 +0000 UTC m=+107.810068846" watchObservedRunningTime="2026-07-16 17:34:24.14910193 +0000 UTC m=+107.810665564" Jul 16 17:34:26.374081 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:26.374024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:26.374555 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:26.374138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:26.374555 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:26.374157 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:34:26.374555 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:26.374219 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:34.374202329 +0000 UTC m=+118.035765943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : configmap "kube-rbac-proxy" not found Jul 16 17:34:26.374555 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:26.374225 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:34:26.374555 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:26.374264 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:34.374253231 +0000 UTC m=+118.035816842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : secret "insights-runtime-extractor-tls" not found Jul 16 17:34:34.443899 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:34.443850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:34.444374 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:34.443910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:34.444374 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:34.443993 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:34:34.444374 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:34.444059 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls podName:fd09d19f-aca3-43a5-ba18-82f2b996dba4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:50.444042632 +0000 UTC m=+134.105606248 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spllq" (UID: "fd09d19f-aca3-43a5-ba18-82f2b996dba4") : secret "insights-runtime-extractor-tls" not found Jul 16 17:34:34.444541 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:34.444472 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd09d19f-aca3-43a5-ba18-82f2b996dba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:43.264449 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.264413 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-6hbd4"] Jul 16 17:34:43.267536 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.267517 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-6hbd4" Jul 16 17:34:43.269990 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.269963 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-sxhfw\"" Jul 16 17:34:43.270362 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.270335 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 17:34:43.270574 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.270549 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 17:34:43.283150 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.283124 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-6hbd4"] Jul 16 17:34:43.316355 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.316322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxjv\" (UniqueName: \"kubernetes.io/projected/aebf3b4e-aaa5-4156-a625-4c344b053213-kube-api-access-vnxjv\") pod \"downloads-68c9cbb497-6hbd4\" (UID: \"aebf3b4e-aaa5-4156-a625-4c344b053213\") " pod="openshift-console/downloads-68c9cbb497-6hbd4" Jul 16 17:34:43.417070 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.417030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxjv\" (UniqueName: \"kubernetes.io/projected/aebf3b4e-aaa5-4156-a625-4c344b053213-kube-api-access-vnxjv\") pod \"downloads-68c9cbb497-6hbd4\" (UID: \"aebf3b4e-aaa5-4156-a625-4c344b053213\") " pod="openshift-console/downloads-68c9cbb497-6hbd4" Jul 16 17:34:43.430883 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.430849 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxjv\" (UniqueName: \"kubernetes.io/projected/aebf3b4e-aaa5-4156-a625-4c344b053213-kube-api-access-vnxjv\") pod \"downloads-68c9cbb497-6hbd4\" (UID: \"aebf3b4e-aaa5-4156-a625-4c344b053213\") " pod="openshift-console/downloads-68c9cbb497-6hbd4" Jul 16 17:34:43.577452 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.577360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-6hbd4" Jul 16 17:34:43.724428 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:43.724380 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-6hbd4"] Jul 16 17:34:43.728640 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:34:43.728605 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebf3b4e_aaa5_4156_a625_4c344b053213.slice/crio-58c40cbaeb183aae82391ab38d0d6b4d6080ffe71bad02674589d414e4eaa045 WatchSource:0}: Error finding container 58c40cbaeb183aae82391ab38d0d6b4d6080ffe71bad02674589d414e4eaa045: Status 404 returned error can't find the container with id 58c40cbaeb183aae82391ab38d0d6b4d6080ffe71bad02674589d414e4eaa045 Jul 16 17:34:44.191432 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:44.191401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-6hbd4" event={"ID":"aebf3b4e-aaa5-4156-a625-4c344b053213","Type":"ContainerStarted","Data":"58c40cbaeb183aae82391ab38d0d6b4d6080ffe71bad02674589d414e4eaa045"} Jul 16 17:34:46.539592 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:46.539548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:34:46.542246 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:46.542205 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8680dbdf-6918-4634-9412-e88acf7f528a-metrics-certs\") pod \"network-metrics-daemon-dmrj5\" (UID: \"8680dbdf-6918-4634-9412-e88acf7f528a\") " pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:34:46.688136 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:46.688103 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-455gg\"" Jul 16 17:34:46.696306 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:46.696289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmrj5" Jul 16 17:34:46.861020 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:46.860885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dmrj5"] Jul 16 17:34:46.863543 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:34:46.863512 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8680dbdf_6918_4634_9412_e88acf7f528a.slice/crio-ffd5e6b685320cdb2aaca6ca92b35bf24717dec67c8a3a7b9675cdcddfa4f44d WatchSource:0}: Error finding container ffd5e6b685320cdb2aaca6ca92b35bf24717dec67c8a3a7b9675cdcddfa4f44d: Status 404 returned error can't find the container with id ffd5e6b685320cdb2aaca6ca92b35bf24717dec67c8a3a7b9675cdcddfa4f44d Jul 16 17:34:47.201568 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:47.201529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dmrj5" event={"ID":"8680dbdf-6918-4634-9412-e88acf7f528a","Type":"ContainerStarted","Data":"ffd5e6b685320cdb2aaca6ca92b35bf24717dec67c8a3a7b9675cdcddfa4f44d"} Jul 16 17:34:48.206266 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.206211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dmrj5" event={"ID":"8680dbdf-6918-4634-9412-e88acf7f528a","Type":"ContainerStarted","Data":"896e9a6032f0556065f966cae9295042396e14bd285cdaa4fc31fc90fc77b4ba"} Jul 16 17:34:48.206720 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.206296 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dmrj5" event={"ID":"8680dbdf-6918-4634-9412-e88acf7f528a","Type":"ContainerStarted","Data":"7db9dd1d9ac693e185adf69be35938833d2872e13b92a16ce1654d8f8272d748"} Jul 16 17:34:48.225604 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.225545 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dmrj5" podStartSLOduration=131.258179013 podStartE2EDuration="2m12.22552668s" totalImagesPullingTime="967.347667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:48.225382715 +0000 UTC m=+131.886946349" watchObservedRunningTime="2026-07-16 17:34:48.22552668 +0000 UTC m=+131.887090314" Jul 16 17:34:48.597647 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.597564 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2"] Jul 16 17:34:48.600791 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.600765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" Jul 16 17:34:48.603097 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.603071 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hw2v6\"" Jul 16 17:34:48.603374 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.603352 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 17:34:48.608255 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.608229 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2"] Jul 16 17:34:48.657463 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.657424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e1f0cc6-05dd-425c-af5e-b69ea95dae97-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-kj5j2\" (UID: \"9e1f0cc6-05dd-425c-af5e-b69ea95dae97\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" Jul 16 17:34:48.758516 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.758474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e1f0cc6-05dd-425c-af5e-b69ea95dae97-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-kj5j2\" (UID: \"9e1f0cc6-05dd-425c-af5e-b69ea95dae97\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" Jul 16 17:34:48.761398 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.761367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e1f0cc6-05dd-425c-af5e-b69ea95dae97-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-kj5j2\" (UID: \"9e1f0cc6-05dd-425c-af5e-b69ea95dae97\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" Jul 16 17:34:48.912592 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:48.912533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" Jul 16 17:34:49.061449 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:49.061418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2"] Jul 16 17:34:49.065303 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:34:49.065275 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1f0cc6_05dd_425c_af5e_b69ea95dae97.slice/crio-e70327bc086b6930e3d8ecf5931e1a2dafd64bd42e08beceae92fe77205ca7bc WatchSource:0}: Error finding container e70327bc086b6930e3d8ecf5931e1a2dafd64bd42e08beceae92fe77205ca7bc: Status 404 returned error can't find the container with id e70327bc086b6930e3d8ecf5931e1a2dafd64bd42e08beceae92fe77205ca7bc Jul 16 17:34:49.211081 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:49.210998 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" event={"ID":"9e1f0cc6-05dd-425c-af5e-b69ea95dae97","Type":"ContainerStarted","Data":"e70327bc086b6930e3d8ecf5931e1a2dafd64bd42e08beceae92fe77205ca7bc"} Jul 16 17:34:50.473427 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:50.473387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:50.476228 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:50.476197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd09d19f-aca3-43a5-ba18-82f2b996dba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spllq\" (UID: \"fd09d19f-aca3-43a5-ba18-82f2b996dba4\") " pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:50.766235 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:50.766151 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxz8n\"" Jul 16 17:34:50.774377 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:50.774349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-spllq" Jul 16 17:34:50.922212 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:50.922167 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-spllq"] Jul 16 17:34:50.925946 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:34:50.925916 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd09d19f_aca3_43a5_ba18_82f2b996dba4.slice/crio-435ffffd381cd18c2124f18cfb96cf64fa642fe038489c635dd96a658fa75028 WatchSource:0}: Error finding container 435ffffd381cd18c2124f18cfb96cf64fa642fe038489c635dd96a658fa75028: Status 404 returned error can't find the container with id 435ffffd381cd18c2124f18cfb96cf64fa642fe038489c635dd96a658fa75028 Jul 16 17:34:51.219511 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.219476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spllq" event={"ID":"fd09d19f-aca3-43a5-ba18-82f2b996dba4","Type":"ContainerStarted","Data":"435ffffd381cd18c2124f18cfb96cf64fa642fe038489c635dd96a658fa75028"} Jul 16 17:34:51.220835 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.220812 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" event={"ID":"9e1f0cc6-05dd-425c-af5e-b69ea95dae97","Type":"ContainerStarted","Data":"4f5547659ecf4c510dabc46e0da7304e48e2b4d3f362b131d2d569caa6dce847"} Jul 16 17:34:51.221029 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.221010 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" Jul 16 17:34:51.225496 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.225472 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" Jul 16 17:34:51.244215 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.244151 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-kj5j2" podStartSLOduration=1.761415377 podStartE2EDuration="3.244131659s" totalImagesPullingTime="1.482716282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:51.243248445 +0000 UTC m=+134.904812080" watchObservedRunningTime="2026-07-16 17:34:51.244131659 +0000 UTC m=+134.905695295" Jul 16 17:34:51.682256 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.682219 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-qvkrm"] Jul 16 17:34:51.686856 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.686832 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.689622 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.689532 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 17:34:51.689622 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.689559 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 17:34:51.690613 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.690461 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 17:34:51.690613 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.690499 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 17:34:51.691356 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.690883 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7ks94\"" Jul 16 17:34:51.691356 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.690900 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 17:34:51.695537 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.695283 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-qvkrm"] Jul 16 17:34:51.782868 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.782832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/de07317d-4b69-4ad5-b00a-a0a4d52308ce-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.783051 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.782942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnvlw\" (UniqueName: \"kubernetes.io/projected/de07317d-4b69-4ad5-b00a-a0a4d52308ce-kube-api-access-gnvlw\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.783051 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.782994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de07317d-4b69-4ad5-b00a-a0a4d52308ce-metrics-client-ca\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.783051 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.783023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de07317d-4b69-4ad5-b00a-a0a4d52308ce-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.883853 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.883820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de07317d-4b69-4ad5-b00a-a0a4d52308ce-metrics-client-ca\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.883853 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.883859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de07317d-4b69-4ad5-b00a-a0a4d52308ce-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.884097 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.883935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/de07317d-4b69-4ad5-b00a-a0a4d52308ce-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.884097 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.883986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnvlw\" (UniqueName: \"kubernetes.io/projected/de07317d-4b69-4ad5-b00a-a0a4d52308ce-kube-api-access-gnvlw\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.884198 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:51.884127 2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 16 17:34:51.884198 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:34:51.884197 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de07317d-4b69-4ad5-b00a-a0a4d52308ce-prometheus-operator-tls podName:de07317d-4b69-4ad5-b00a-a0a4d52308ce nodeName:}" failed. No retries permitted until 2026-07-16 17:34:52.384176547 +0000 UTC m=+136.045740171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/de07317d-4b69-4ad5-b00a-a0a4d52308ce-prometheus-operator-tls") pod "prometheus-operator-64785b6675-qvkrm" (UID: "de07317d-4b69-4ad5-b00a-a0a4d52308ce") : secret "prometheus-operator-tls" not found Jul 16 17:34:51.884648 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.884626 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de07317d-4b69-4ad5-b00a-a0a4d52308ce-metrics-client-ca\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.886739 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.886712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de07317d-4b69-4ad5-b00a-a0a4d52308ce-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:51.896532 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:51.896503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnvlw\" (UniqueName: \"kubernetes.io/projected/de07317d-4b69-4ad5-b00a-a0a4d52308ce-kube-api-access-gnvlw\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:52.389670 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:52.389554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/de07317d-4b69-4ad5-b00a-a0a4d52308ce-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:52.392609 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:52.392569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/de07317d-4b69-4ad5-b00a-a0a4d52308ce-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-qvkrm\" (UID: \"de07317d-4b69-4ad5-b00a-a0a4d52308ce\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:52.599048 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:52.599009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" Jul 16 17:34:52.758902 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:52.758866 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-qvkrm"] Jul 16 17:34:52.762286 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:34:52.762259 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde07317d_4b69_4ad5_b00a_a0a4d52308ce.slice/crio-938a2048e9e32cc436cce3497ee5b5b4a83c9cb2e771ba94737d9cb5b3fb2f18 WatchSource:0}: Error finding container 938a2048e9e32cc436cce3497ee5b5b4a83c9cb2e771ba94737d9cb5b3fb2f18: Status 404 returned error can't find the container with id 938a2048e9e32cc436cce3497ee5b5b4a83c9cb2e771ba94737d9cb5b3fb2f18 Jul 16 17:34:53.238598 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:34:53.238552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" event={"ID":"de07317d-4b69-4ad5-b00a-a0a4d52308ce","Type":"ContainerStarted","Data":"938a2048e9e32cc436cce3497ee5b5b4a83c9cb2e771ba94737d9cb5b3fb2f18"} Jul 16 17:35:04.277716 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:04.277649 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-6hbd4" event={"ID":"aebf3b4e-aaa5-4156-a625-4c344b053213","Type":"ContainerStarted","Data":"0832fd33b0f5371913c323a6532463c6084e067b49523f3a67e466e8f67bc2ec"} Jul 16 17:35:04.278266 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:04.277871 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-6hbd4" Jul 16 17:35:04.280096 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:04.280066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" event={"ID":"de07317d-4b69-4ad5-b00a-a0a4d52308ce","Type":"ContainerStarted","Data":"90430c52b552f5e729cf5e826e28d097c83d09c5b2afd17b97666af3adad2bcc"} Jul 16 17:35:04.280219 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:04.280119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" event={"ID":"de07317d-4b69-4ad5-b00a-a0a4d52308ce","Type":"ContainerStarted","Data":"a7991b30bb9701d2dec25d180b443ec2f728891bab56a9d8655d3947b7bbbb17"} Jul 16 17:35:04.281610 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:04.281590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spllq" event={"ID":"fd09d19f-aca3-43a5-ba18-82f2b996dba4","Type":"ContainerStarted","Data":"2b4a88924f1356e36ce33308ae339e413fda102f3c11e8f165e5741ed5f9dca4"} Jul 16 17:35:04.293404 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:04.293378 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-6hbd4" Jul 16 17:35:04.311503 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:04.311445 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-6hbd4" podStartSLOduration=1.224683913 podStartE2EDuration="21.311424491s" totalImagesPullingTime="20.086740578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:04.310720374 +0000 UTC m=+147.972284000" watchObservedRunningTime="2026-07-16 17:35:04.311424491 +0000 UTC m=+147.972988126" Jul 16 17:35:04.377735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:04.376845 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-qvkrm" podStartSLOduration=2.373834501 podStartE2EDuration="13.376816727s" totalImagesPullingTime="11.002982226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:04.341955938 +0000 UTC m=+148.003519572" watchObservedRunningTime="2026-07-16 17:35:04.376816727 +0000 UTC m=+148.038380361" Jul 16 17:35:05.457443 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:05.456917 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846b6cfbb5-9jhz2"] Jul 16 17:35:05.457443 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:35:05.457386 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" podUID="4a8a4c22-ae74-44b4-87bb-807398b17251" Jul 16 17:35:06.287901 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.287463 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-952wv"] Jul 16 17:35:06.293398 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.293370 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:35:06.351781 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.351749 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-952wv"] Jul 16 17:35:06.351962 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.351810 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-7zhkn"] Jul 16 17:35:06.352032 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.351980 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.355655 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.355535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:35:06.366305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.366282 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 17:35:06.366546 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.366529 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nc94m\"" Jul 16 17:35:06.366741 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.366713 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 17:35:06.369163 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.368531 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-7zhkn"] Jul 16 17:35:06.369163 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.368590 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9rlxm"] Jul 16 17:35:06.369950 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.369920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.374635 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.374615 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 17:35:06.375412 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.374996 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 17:35:06.375412 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.375212 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 17:35:06.375412 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.375243 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wdpdw\"" Jul 16 17:35:06.380951 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.380926 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.386265 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.385340 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kmqhb\"" Jul 16 17:35:06.386265 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.385771 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 17:35:06.386265 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.385998 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 17:35:06.386265 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.386169 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 17:35:06.414847 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.414814 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-trusted-ca\") pod \"4a8a4c22-ae74-44b4-87bb-807398b17251\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " Jul 16 17:35:06.415024 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.414872 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-image-registry-private-configuration\") pod \"4a8a4c22-ae74-44b4-87bb-807398b17251\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " Jul 16 17:35:06.415024 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.414907 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-bound-sa-token\") pod \"4a8a4c22-ae74-44b4-87bb-807398b17251\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " Jul 16 17:35:06.415024 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.414934 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-certificates\") pod \"4a8a4c22-ae74-44b4-87bb-807398b17251\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " Jul 16 17:35:06.415024 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.414963 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-installation-pull-secrets\") pod \"4a8a4c22-ae74-44b4-87bb-807398b17251\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " Jul 16 17:35:06.415248 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.415029 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a8a4c22-ae74-44b4-87bb-807398b17251-ca-trust-extracted\") pod \"4a8a4c22-ae74-44b4-87bb-807398b17251\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " Jul 16 17:35:06.415248 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.415083 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rclqz\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-kube-api-access-rclqz\") pod \"4a8a4c22-ae74-44b4-87bb-807398b17251\" (UID: \"4a8a4c22-ae74-44b4-87bb-807398b17251\") " Jul 16 17:35:06.415248 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.415198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c65fe41-57a0-4741-bcd0-3eef39882bde-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.415399 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.415263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c65fe41-57a0-4741-bcd0-3eef39882bde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.415399 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.415302 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4a8a4c22-ae74-44b4-87bb-807398b17251" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:35:06.415399 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.415321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c65fe41-57a0-4741-bcd0-3eef39882bde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.415399 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.415395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r997f\" (UniqueName: \"kubernetes.io/projected/9c65fe41-57a0-4741-bcd0-3eef39882bde-kube-api-access-r997f\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.415601 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.415448 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-trusted-ca\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:35:06.415655 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.415633 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8a4c22-ae74-44b4-87bb-807398b17251-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4a8a4c22-ae74-44b4-87bb-807398b17251" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:35:06.416312 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.416022 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4a8a4c22-ae74-44b4-87bb-807398b17251" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:35:06.419277 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.419226 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4a8a4c22-ae74-44b4-87bb-807398b17251" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:35:06.419277 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.419251 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-kube-api-access-rclqz" (OuterVolumeSpecName: "kube-api-access-rclqz") pod "4a8a4c22-ae74-44b4-87bb-807398b17251" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251"). InnerVolumeSpecName "kube-api-access-rclqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:35:06.419515 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.419451 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4a8a4c22-ae74-44b4-87bb-807398b17251" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:35:06.419577 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.419538 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4a8a4c22-ae74-44b4-87bb-807398b17251" (UID: "4a8a4c22-ae74-44b4-87bb-807398b17251"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:35:06.516924 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.516884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r997f\" (UniqueName: \"kubernetes.io/projected/9c65fe41-57a0-4741-bcd0-3eef39882bde-kube-api-access-r997f\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.516941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.516966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-sys\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.516988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-wtmp\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-root\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71c818d1-4c62-4acf-aa66-f0bd065eee9b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517179 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c65fe41-57a0-4741-bcd0-3eef39882bde-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vc4h\" (UniqueName: \"kubernetes.io/projected/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-api-access-7vc4h\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.517373 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-tls\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c65fe41-57a0-4741-bcd0-3eef39882bde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-metrics-client-ca\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c65fe41-57a0-4741-bcd0-3eef39882bde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71c818d1-4c62-4acf-aa66-f0bd065eee9b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66fl\" (UniqueName: \"kubernetes.io/projected/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-kube-api-access-v66fl\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-textfile\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517627 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rclqz\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-kube-api-access-rclqz\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517645 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-image-registry-private-configuration\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517659 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-bound-sa-token\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517673 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-certificates\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517686 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a8a4c22-ae74-44b4-87bb-807398b17251-installation-pull-secrets\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:35:06.517970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.517722 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a8a4c22-ae74-44b4-87bb-807398b17251-ca-trust-extracted\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:35:06.518561 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.518063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c65fe41-57a0-4741-bcd0-3eef39882bde-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.520763 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.520692 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c65fe41-57a0-4741-bcd0-3eef39882bde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.521125 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.521102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c65fe41-57a0-4741-bcd0-3eef39882bde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.528254 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.528205 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r997f\" (UniqueName: \"kubernetes.io/projected/9c65fe41-57a0-4741-bcd0-3eef39882bde-kube-api-access-r997f\") pod \"openshift-state-metrics-84bd68b896-952wv\" (UID: \"9c65fe41-57a0-4741-bcd0-3eef39882bde\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.618780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.618849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vc4h\" (UniqueName: \"kubernetes.io/projected/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-api-access-7vc4h\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.618893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-tls\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.618937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-metrics-client-ca\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.618964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71c818d1-4c62-4acf-aa66-f0bd065eee9b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.618991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66fl\" (UniqueName: \"kubernetes.io/projected/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-kube-api-access-v66fl\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.619036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.619064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-textfile\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.619107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.619130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-sys\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.619154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-wtmp\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.619207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.624304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.619232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-root\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.619264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.619289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71c818d1-4c62-4acf-aa66-f0bd065eee9b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.620073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71c818d1-4c62-4acf-aa66-f0bd065eee9b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.620571 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:35:06.620985 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:35:06.621053 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-tls podName:bbdfa7f8-fa05-48c7-9cf3-ac75acff0006 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:07.121034259 +0000 UTC m=+150.782597874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-tls") pod "node-exporter-9rlxm" (UID: "bbdfa7f8-fa05-48c7-9cf3-ac75acff0006") : secret "node-exporter-tls" not found Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.621685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-metrics-client-ca\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.621994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71c818d1-4c62-4acf-aa66-f0bd065eee9b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.622901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.623164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-textfile\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.623908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-sys\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.624784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-root\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.625439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.624931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-wtmp\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.629168 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.628994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.631236 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.631194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.633607 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.633566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vc4h\" (UniqueName: \"kubernetes.io/projected/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-api-access-7vc4h\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.634589 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.634547 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71c818d1-4c62-4acf-aa66-f0bd065eee9b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-7zhkn\" (UID: \"71c818d1-4c62-4acf-aa66-f0bd065eee9b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:06.638654 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.638630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66fl\" (UniqueName: \"kubernetes.io/projected/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-kube-api-access-v66fl\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:06.670489 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.670452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" Jul 16 17:35:06.684973 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:06.684945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" Jul 16 17:35:07.124403 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.124359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-tls\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:07.127389 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.127331 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbdfa7f8-fa05-48c7-9cf3-ac75acff0006-node-exporter-tls\") pod \"node-exporter-9rlxm\" (UID: \"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006\") " pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:07.295167 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.295128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9rlxm" Jul 16 17:35:07.299404 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.299376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846b6cfbb5-9jhz2" Jul 16 17:35:07.458428 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.458347 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846b6cfbb5-9jhz2"] Jul 16 17:35:07.464905 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.464871 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-846b6cfbb5-9jhz2"] Jul 16 17:35:07.510996 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.510961 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:35:07.529376 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.529325 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a8a4c22-ae74-44b4-87bb-807398b17251-registry-tls\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:35:07.564621 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.563480 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:35:07.564621 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.563672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.569468 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.569439 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 17:35:07.569878 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.569857 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 17:35:07.569997 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.569890 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 17:35:07.569997 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.569914 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 17:35:07.570311 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.570292 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 17:35:07.570587 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.570567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gbl27\"" Jul 16 17:35:07.570685 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.570594 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 17:35:07.571338 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.571305 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 17:35:07.573506 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.573483 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 17:35:07.574468 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.574450 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 17:35:07.629954 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.629918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630124 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.629961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630124 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.629989 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630124 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-out\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630124 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630325 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630325 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630325 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-web-config\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630430 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630430 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cvdf\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-kube-api-access-6cvdf\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630430 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630602 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.630602 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.630454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.731986 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.731901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-web-config\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.731986 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.731955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.732210 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.732083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cvdf\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-kube-api-access-6cvdf\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.732210 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.732139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.732317 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.732262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.732371 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.732314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.732371 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.732355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.732467 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.732391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.732467 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.732421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.732839 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.732803 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.733008 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.732981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.733102 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.733044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-out\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.733102 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.733078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.733204 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.733107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.733204 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.733153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.735554 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.735223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.735554 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.735344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.736078 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.736031 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.736325 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.736278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-web-config\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.736431 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.736415 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-out\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.736994 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.736952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.737477 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.737400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.737961 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.737917 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.738521 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.738478 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.738626 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.738607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.745990 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.745963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cvdf\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-kube-api-access-6cvdf\") pod \"alertmanager-main-0\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:07.877477 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:07.877439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:35:08.771147 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:08.771113 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a8a4c22-ae74-44b4-87bb-807398b17251" path="/var/lib/kubelet/pods/4a8a4c22-ae74-44b4-87bb-807398b17251/volumes" Jul 16 17:35:08.840253 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:35:08.840183 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdfa7f8_fa05_48c7_9cf3_ac75acff0006.slice/crio-bae7930f032353a8b20c4cdae76da0d3bf54feddc58efbaaea71ad1f0fd3b422 WatchSource:0}: Error finding container bae7930f032353a8b20c4cdae76da0d3bf54feddc58efbaaea71ad1f0fd3b422: Status 404 returned error can't find the container with id bae7930f032353a8b20c4cdae76da0d3bf54feddc58efbaaea71ad1f0fd3b422 Jul 16 17:35:09.010929 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:09.010888 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-7zhkn"] Jul 16 17:35:09.015077 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:35:09.015047 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c818d1_4c62_4acf_aa66_f0bd065eee9b.slice/crio-c75ed66cff93004830bc859177ab9b6bac7c7b772603e874c5ec61b8052fd4ff WatchSource:0}: Error finding container c75ed66cff93004830bc859177ab9b6bac7c7b772603e874c5ec61b8052fd4ff: Status 404 returned error can't find the container with id c75ed66cff93004830bc859177ab9b6bac7c7b772603e874c5ec61b8052fd4ff Jul 16 17:35:09.242069 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:09.242040 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-952wv"] Jul 16 17:35:09.254888 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:35:09.254856 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c65fe41_57a0_4741_bcd0_3eef39882bde.slice/crio-97437f5475d96d291b9ecfe278f7d50d721311c7b57e0b9098e94d44e980eaaf WatchSource:0}: Error finding container 97437f5475d96d291b9ecfe278f7d50d721311c7b57e0b9098e94d44e980eaaf: Status 404 returned error can't find the container with id 97437f5475d96d291b9ecfe278f7d50d721311c7b57e0b9098e94d44e980eaaf Jul 16 17:35:09.256409 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:09.256322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:35:09.266641 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:35:09.266559 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ce6193_1165_4e1e_a5a0_e11d4631f55c.slice/crio-3a4ab0ac9712bedce52e802bb73e5ff6704da42152b135e69cb2f24ee85c35d0 WatchSource:0}: Error finding container 3a4ab0ac9712bedce52e802bb73e5ff6704da42152b135e69cb2f24ee85c35d0: Status 404 returned error can't find the container with id 3a4ab0ac9712bedce52e802bb73e5ff6704da42152b135e69cb2f24ee85c35d0 Jul 16 17:35:09.311829 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:09.310213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerStarted","Data":"3a4ab0ac9712bedce52e802bb73e5ff6704da42152b135e69cb2f24ee85c35d0"} Jul 16 17:35:09.313359 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:09.313320 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rlxm" event={"ID":"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006","Type":"ContainerStarted","Data":"bae7930f032353a8b20c4cdae76da0d3bf54feddc58efbaaea71ad1f0fd3b422"} Jul 16 17:35:09.314486 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:09.314463 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" event={"ID":"9c65fe41-57a0-4741-bcd0-3eef39882bde","Type":"ContainerStarted","Data":"97437f5475d96d291b9ecfe278f7d50d721311c7b57e0b9098e94d44e980eaaf"} Jul 16 17:35:09.328032 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:09.325850 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" event={"ID":"71c818d1-4c62-4acf-aa66-f0bd065eee9b","Type":"ContainerStarted","Data":"c75ed66cff93004830bc859177ab9b6bac7c7b772603e874c5ec61b8052fd4ff"} Jul 16 17:35:10.336029 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.335940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" event={"ID":"9c65fe41-57a0-4741-bcd0-3eef39882bde","Type":"ContainerStarted","Data":"1f1fba6387bbf412ae45627966df7f08584990f78f27e5de3d923e83b42f2d0f"} Jul 16 17:35:10.336029 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.336001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" event={"ID":"9c65fe41-57a0-4741-bcd0-3eef39882bde","Type":"ContainerStarted","Data":"01635ecdd6e5c7f74462275c4b195850fd386c95ef1707a067391b96ee85428d"} Jul 16 17:35:10.339392 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.339361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spllq" event={"ID":"fd09d19f-aca3-43a5-ba18-82f2b996dba4","Type":"ContainerStarted","Data":"4e121ae1b08989d456e95bd433f86b4b3c561a9cae861d391ca6848d7f79d1d9"} Jul 16 17:35:10.745835 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.745124 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-557cf879c5-tm4hc"] Jul 16 17:35:10.783302 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.781953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.787322 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.787019 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 17:35:10.787974 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.787954 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 17:35:10.788279 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.788263 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 17:35:10.788492 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.788461 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-p65cd\"" Jul 16 17:35:10.788663 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.788648 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-88krqg9ohnb0r\"" Jul 16 17:35:10.788997 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.788901 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 17:35:10.792647 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.792594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-557cf879c5-tm4hc"] Jul 16 17:35:10.866246 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.866203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/edfab048-9ffd-4be9-96f1-0909deb3a0a4-secret-metrics-server-tls\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.866426 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.866305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/edfab048-9ffd-4be9-96f1-0909deb3a0a4-secret-metrics-server-client-certs\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.866426 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.866338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftqg\" (UniqueName: \"kubernetes.io/projected/edfab048-9ffd-4be9-96f1-0909deb3a0a4-kube-api-access-sftqg\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.866426 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.866374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edfab048-9ffd-4be9-96f1-0909deb3a0a4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.866596 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.866429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/edfab048-9ffd-4be9-96f1-0909deb3a0a4-metrics-server-audit-profiles\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.866596 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.866494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/edfab048-9ffd-4be9-96f1-0909deb3a0a4-audit-log\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.866596 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.866523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/edfab048-9ffd-4be9-96f1-0909deb3a0a4-client-ca-bundle\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.967223 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.967182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/edfab048-9ffd-4be9-96f1-0909deb3a0a4-secret-metrics-server-tls\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.967430 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.967287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/edfab048-9ffd-4be9-96f1-0909deb3a0a4-secret-metrics-server-client-certs\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.967430 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.967317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftqg\" (UniqueName: \"kubernetes.io/projected/edfab048-9ffd-4be9-96f1-0909deb3a0a4-kube-api-access-sftqg\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.967430 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.967351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edfab048-9ffd-4be9-96f1-0909deb3a0a4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.967430 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.967388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/edfab048-9ffd-4be9-96f1-0909deb3a0a4-metrics-server-audit-profiles\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.967430 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.967417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/edfab048-9ffd-4be9-96f1-0909deb3a0a4-audit-log\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.967716 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.967439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/edfab048-9ffd-4be9-96f1-0909deb3a0a4-client-ca-bundle\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.968674 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.968329 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/edfab048-9ffd-4be9-96f1-0909deb3a0a4-audit-log\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.968851 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.968826 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edfab048-9ffd-4be9-96f1-0909deb3a0a4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.969921 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.969066 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/edfab048-9ffd-4be9-96f1-0909deb3a0a4-metrics-server-audit-profiles\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.972248 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.971211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/edfab048-9ffd-4be9-96f1-0909deb3a0a4-secret-metrics-server-tls\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.972248 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.971951 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/edfab048-9ffd-4be9-96f1-0909deb3a0a4-client-ca-bundle\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.972402 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.972276 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/edfab048-9ffd-4be9-96f1-0909deb3a0a4-secret-metrics-server-client-certs\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:10.987343 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:10.987287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftqg\" (UniqueName: \"kubernetes.io/projected/edfab048-9ffd-4be9-96f1-0909deb3a0a4-kube-api-access-sftqg\") pod \"metrics-server-557cf879c5-tm4hc\" (UID: \"edfab048-9ffd-4be9-96f1-0909deb3a0a4\") " pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:11.099644 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:11.099531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:13.369538 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:13.369510 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-557cf879c5-tm4hc"] Jul 16 17:35:14.360749 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.360688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" event={"ID":"9c65fe41-57a0-4741-bcd0-3eef39882bde","Type":"ContainerStarted","Data":"ef4d4bcc586c72c2cdb1a315d19cd6cd8d9d7d6eba1e9e3d17b44f4de353233b"} Jul 16 17:35:14.363013 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.362984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" event={"ID":"71c818d1-4c62-4acf-aa66-f0bd065eee9b","Type":"ContainerStarted","Data":"f91127afc5f95b9f018c684480f2509d3de9ff80f922d5f989ca8ec68d6b5e6d"} Jul 16 17:35:14.363149 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.363042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" event={"ID":"71c818d1-4c62-4acf-aa66-f0bd065eee9b","Type":"ContainerStarted","Data":"cc40fd04b6e7a053ee313d9f3a8f8c61d5a985eb521238aa0d15d1d7f41df2f7"} Jul 16 17:35:14.363149 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.363068 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" event={"ID":"71c818d1-4c62-4acf-aa66-f0bd065eee9b","Type":"ContainerStarted","Data":"2297791206bbb914b168c3734256dcbabe04b303786b953c5099889b3e620f4e"} Jul 16 17:35:14.364327 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.364302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" event={"ID":"edfab048-9ffd-4be9-96f1-0909deb3a0a4","Type":"ContainerStarted","Data":"80ceed52572a6928f0a8b53fda9f19187bd7b8be9a6165f31282f103da73a907"} Jul 16 17:35:14.365781 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.365760 2576 generic.go:356] "Generic (PLEG): container finished" podID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerID="0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6" exitCode=0 Jul 16 17:35:14.365876 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.365845 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerDied","Data":"0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6"} Jul 16 17:35:14.367689 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.367659 2576 generic.go:356] "Generic (PLEG): container finished" podID="bbdfa7f8-fa05-48c7-9cf3-ac75acff0006" containerID="c69dc3cf9d800855aa90783f8ef9120af9099a5d9e78713b8193518ffd4249c1" exitCode=0 Jul 16 17:35:14.367798 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.367724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rlxm" event={"ID":"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006","Type":"ContainerDied","Data":"c69dc3cf9d800855aa90783f8ef9120af9099a5d9e78713b8193518ffd4249c1"} Jul 16 17:35:14.390806 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.390689 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-952wv" podStartSLOduration=4.664716711 podStartE2EDuration="8.390671917s" totalImagesPullingTime="3.725955206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:14.389308191 +0000 UTC m=+158.050871824" watchObservedRunningTime="2026-07-16 17:35:14.390671917 +0000 UTC m=+158.052235582" Jul 16 17:35:14.478092 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:14.478012 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-7zhkn" podStartSLOduration=4.465511709 podStartE2EDuration="8.477991441s" totalImagesPullingTime="4.012479732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:14.476238723 +0000 UTC m=+158.137802360" watchObservedRunningTime="2026-07-16 17:35:14.477991441 +0000 UTC m=+158.139555073" Jul 16 17:35:14.791663 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:35:14.791611 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xtq5p" podUID="8ce9162f-6d4d-44ec-a1ea-4d5e2652c758" Jul 16 17:35:15.372855 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:15.372829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xtq5p" Jul 16 17:35:16.379279 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:16.379238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spllq" event={"ID":"fd09d19f-aca3-43a5-ba18-82f2b996dba4","Type":"ContainerStarted","Data":"313fb5d61521911020f03880100459072545884ea328c1edbb20675ac53188fb"} Jul 16 17:35:16.381632 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:16.381606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rlxm" event={"ID":"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006","Type":"ContainerStarted","Data":"b3e50b16599ec2cd28f2538589c6deff290e93d009e66a39906092232a3701ed"} Jul 16 17:35:16.381764 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:16.381658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rlxm" event={"ID":"bbdfa7f8-fa05-48c7-9cf3-ac75acff0006","Type":"ContainerStarted","Data":"720a3fd706ea96648100da70b9933fabd65fffbe7c260690591af2183d9fba91"} Jul 16 17:35:16.404438 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:16.404370 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-spllq" podStartSLOduration=33.597132648 podStartE2EDuration="58.404350842s" totalImagesPullingTime="24.807218194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:16.404280639 +0000 UTC m=+160.065844273" watchObservedRunningTime="2026-07-16 17:35:16.404350842 +0000 UTC m=+160.065914476" Jul 16 17:35:16.431367 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:16.431316 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9rlxm" podStartSLOduration=5.957523444 podStartE2EDuration="10.431300655s" totalImagesPullingTime="4.35044174s" totalInitContainerRuntime="123.335471ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:16.42961176 +0000 UTC m=+160.091175408" watchObservedRunningTime="2026-07-16 17:35:16.431300655 +0000 UTC m=+160.092864288" Jul 16 17:35:18.393233 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:18.393199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerStarted","Data":"9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f"} Jul 16 17:35:18.394622 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:18.394593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" event={"ID":"edfab048-9ffd-4be9-96f1-0909deb3a0a4","Type":"ContainerStarted","Data":"3905d4d1c83bab1b1b47ce34a5ea94837fb2cc946c5e9f993d3b182aae5200d8"} Jul 16 17:35:18.430636 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:18.430581 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" podStartSLOduration=4.50600548 podStartE2EDuration="8.430564148s" totalImagesPullingTime="3.924558668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:18.42940519 +0000 UTC m=+162.090968837" watchObservedRunningTime="2026-07-16 17:35:18.430564148 +0000 UTC m=+162.092127781" Jul 16 17:35:19.406790 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:19.406745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerStarted","Data":"46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1"} Jul 16 17:35:19.407233 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:19.406842 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerStarted","Data":"b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2"} Jul 16 17:35:19.407233 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:19.406870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerStarted","Data":"2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206"} Jul 16 17:35:19.407233 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:19.406894 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerStarted","Data":"08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d"} Jul 16 17:35:19.755801 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:19.755687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:35:19.758024 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:19.757998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9162f-6d4d-44ec-a1ea-4d5e2652c758-metrics-tls\") pod \"dns-default-xtq5p\" (UID: \"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758\") " pod="openshift-dns/dns-default-xtq5p" Jul 16 17:35:19.879607 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:19.879579 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7tn9x\"" Jul 16 17:35:19.883909 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:19.883893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xtq5p" Jul 16 17:35:20.058463 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:20.058436 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xtq5p"] Jul 16 17:35:20.059923 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:35:20.059894 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce9162f_6d4d_44ec_a1ea_4d5e2652c758.slice/crio-38266cee3c887ae9989c77328daf4a4d71be962200779967bc278faf15b14dd2 WatchSource:0}: Error finding container 38266cee3c887ae9989c77328daf4a4d71be962200779967bc278faf15b14dd2: Status 404 returned error can't find the container with id 38266cee3c887ae9989c77328daf4a4d71be962200779967bc278faf15b14dd2 Jul 16 17:35:20.413089 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:20.413054 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xtq5p" event={"ID":"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758","Type":"ContainerStarted","Data":"38266cee3c887ae9989c77328daf4a4d71be962200779967bc278faf15b14dd2"} Jul 16 17:35:21.420668 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:21.420635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerStarted","Data":"a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661"} Jul 16 17:35:21.458661 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:21.458595 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.397329231 podStartE2EDuration="14.458576951s" totalImagesPullingTime="10.015803999s" totalInitContainerRuntime="45.443721ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:21.457458955 +0000 UTC m=+165.119022598" watchObservedRunningTime="2026-07-16 17:35:21.458576951 +0000 UTC m=+165.120140584" Jul 16 17:35:23.429269 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:23.429234 2576 generic.go:356] "Generic (PLEG): container finished" podID="6feb2440-11a1-4c36-9d88-bcd0e715062a" containerID="fa351f78287fb2c6a1acca50b57636c099eb3355ad998c6b096161765823e667" exitCode=0 Jul 16 17:35:23.429735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:23.429299 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" event={"ID":"6feb2440-11a1-4c36-9d88-bcd0e715062a","Type":"ContainerDied","Data":"fa351f78287fb2c6a1acca50b57636c099eb3355ad998c6b096161765823e667"} Jul 16 17:35:23.429735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:23.429689 2576 scope.go:117] "RemoveContainer" containerID="fa351f78287fb2c6a1acca50b57636c099eb3355ad998c6b096161765823e667" Jul 16 17:35:24.440986 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:24.440950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-x48rl" event={"ID":"6feb2440-11a1-4c36-9d88-bcd0e715062a","Type":"ContainerStarted","Data":"df1b01c7075742a741866c155c076031e25832259bcae9c72dac5e3dd0c816f9"} Jul 16 17:35:24.442333 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:24.442309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xtq5p" event={"ID":"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758","Type":"ContainerStarted","Data":"f996aa6f5c28fef384acb498fbd81392894a1209fa3edb4dba2c6ea95734c494"} Jul 16 17:35:25.447787 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:25.447751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xtq5p" event={"ID":"8ce9162f-6d4d-44ec-a1ea-4d5e2652c758","Type":"ContainerStarted","Data":"8c1ec3cff281ea337175872a8fddcce785defd85e91854b9658b2bdcb85f6d83"} Jul 16 17:35:25.448243 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:25.447892 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xtq5p" Jul 16 17:35:31.100345 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:31.100259 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:31.100345 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:31.100323 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:35.454518 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:35.454482 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xtq5p" Jul 16 17:35:35.479371 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:35.479314 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xtq5p" podStartSLOduration=140.332127659 podStartE2EDuration="2m24.479296612s" totalImagesPullingTime="4.147168953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:25.476449781 +0000 UTC m=+169.138013413" watchObservedRunningTime="2026-07-16 17:35:35.479296612 +0000 UTC m=+179.140860244" Jul 16 17:35:51.105309 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:51.105276 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:35:51.109221 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:35:51.109195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-557cf879c5-tm4hc" Jul 16 17:36:26.721509 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:26.721469 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:36:26.722089 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:26.721805 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="alertmanager" containerID="cri-o://9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f" gracePeriod=120 Jul 16 17:36:26.722089 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:26.721859 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy-metric" containerID="cri-o://46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1" gracePeriod=120 Jul 16 17:36:26.722089 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:26.721880 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="prom-label-proxy" containerID="cri-o://a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661" gracePeriod=120 Jul 16 17:36:26.722089 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:26.721917 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy-web" containerID="cri-o://2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206" gracePeriod=120 Jul 16 17:36:26.722089 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:26.721936 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="config-reloader" containerID="cri-o://08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d" gracePeriod=120 Jul 16 17:36:26.722089 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:26.721997 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy" containerID="cri-o://b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2" gracePeriod=120 Jul 16 17:36:27.684760 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684730 2576 generic.go:356] "Generic (PLEG): container finished" podID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerID="a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661" exitCode=0 Jul 16 17:36:27.684760 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684756 2576 generic.go:356] "Generic (PLEG): container finished" podID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerID="46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1" exitCode=0 Jul 16 17:36:27.684760 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684766 2576 generic.go:356] "Generic (PLEG): container finished" podID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerID="b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2" exitCode=0 Jul 16 17:36:27.684991 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684773 2576 generic.go:356] "Generic (PLEG): container finished" podID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerID="08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d" exitCode=0 Jul 16 17:36:27.684991 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684780 2576 generic.go:356] "Generic (PLEG): container finished" podID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerID="9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f" exitCode=0 Jul 16 17:36:27.684991 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684799 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerDied","Data":"a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661"} Jul 16 17:36:27.684991 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerDied","Data":"46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1"} Jul 16 17:36:27.684991 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerDied","Data":"b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2"} Jul 16 17:36:27.684991 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerDied","Data":"08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d"} Jul 16 17:36:27.684991 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.684939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerDied","Data":"9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f"} Jul 16 17:36:27.963028 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:27.963001 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.118497 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118462 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-volume\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.118659 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118515 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-out\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.118659 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118534 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-main-tls\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.118659 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118560 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-tls-assets\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.118659 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118585 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-cluster-tls-config\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.118659 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118602 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cvdf\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-kube-api-access-6cvdf\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.118659 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118625 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.118659 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118649 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-trusted-ca-bundle\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.119055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118685 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-main-db\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.119055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118739 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-web-config\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.119055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118799 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-metrics-client-ca\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.119055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118831 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.119055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.118862 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\" (UID: \"02ce6193-1165-4e1e-a5a0-e11d4631f55c\") " Jul 16 17:36:28.119311 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.119161 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:36:28.119453 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.119425 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:36:28.119945 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.119882 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:36:28.121348 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.121309 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-out" (OuterVolumeSpecName: "config-out") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:36:28.121970 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.121940 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-volume" (OuterVolumeSpecName: "config-volume") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:28.122082 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.122064 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:28.122185 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.122155 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:28.122575 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.122542 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:36:28.122575 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.122546 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-kube-api-access-6cvdf" (OuterVolumeSpecName: "kube-api-access-6cvdf") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "kube-api-access-6cvdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:36:28.122744 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.122649 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:28.123475 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.123447 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:28.125887 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.125867 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:28.132938 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.132914 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-web-config" (OuterVolumeSpecName: "web-config") pod "02ce6193-1165-4e1e-a5a0-e11d4631f55c" (UID: "02ce6193-1165-4e1e-a5a0-e11d4631f55c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:28.220451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220359 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220390 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220403 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-volume\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220412 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-config-out\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220420 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-main-tls\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220430 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-tls-assets\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220439 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-cluster-tls-config\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220448 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cvdf\" (UniqueName: \"kubernetes.io/projected/02ce6193-1165-4e1e-a5a0-e11d4631f55c-kube-api-access-6cvdf\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220457 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220895 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220468 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220895 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220477 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/02ce6193-1165-4e1e-a5a0-e11d4631f55c-alertmanager-main-db\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220895 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220485 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02ce6193-1165-4e1e-a5a0-e11d4631f55c-web-config\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.220895 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.220493 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ce6193-1165-4e1e-a5a0-e11d4631f55c-metrics-client-ca\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:36:28.691815 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.691783 2576 generic.go:356] "Generic (PLEG): container finished" podID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerID="2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206" exitCode=0 Jul 16 17:36:28.692012 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.691839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerDied","Data":"2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206"} Jul 16 17:36:28.692012 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.691867 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.692012 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.691898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02ce6193-1165-4e1e-a5a0-e11d4631f55c","Type":"ContainerDied","Data":"3a4ab0ac9712bedce52e802bb73e5ff6704da42152b135e69cb2f24ee85c35d0"} Jul 16 17:36:28.692012 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.691936 2576 scope.go:117] "RemoveContainer" containerID="a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661" Jul 16 17:36:28.700015 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.699989 2576 scope.go:117] "RemoveContainer" containerID="46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1" Jul 16 17:36:28.707211 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.707194 2576 scope.go:117] "RemoveContainer" containerID="b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2" Jul 16 17:36:28.719162 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.719140 2576 scope.go:117] "RemoveContainer" containerID="2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206" Jul 16 17:36:28.726056 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.726031 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:36:28.726814 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.726790 2576 scope.go:117] "RemoveContainer" containerID="08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d" Jul 16 17:36:28.734043 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.734021 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:36:28.735048 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.735030 2576 scope.go:117] "RemoveContainer" containerID="9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f" Jul 16 17:36:28.742137 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.742119 2576 scope.go:117] "RemoveContainer" containerID="0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6" Jul 16 17:36:28.749412 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.749395 2576 scope.go:117] "RemoveContainer" containerID="a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661" Jul 16 17:36:28.749711 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:36:28.749679 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661\": container with ID starting with a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661 not found: ID does not exist" containerID="a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661" Jul 16 17:36:28.749869 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.749823 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661"} err="failed to get container status \"a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661\": rpc error: code = NotFound desc = could not find container \"a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661\": container with ID starting with a294bfde06d1ca715d500bfa2e2df9c2398279b4e8c27fcf2681e74a9d823661 not found: ID does not exist" Jul 16 17:36:28.749869 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.749870 2576 scope.go:117] "RemoveContainer" containerID="46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1" Jul 16 17:36:28.750107 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:36:28.750093 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1\": container with ID starting with 46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1 not found: ID does not exist" containerID="46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1" Jul 16 17:36:28.750155 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.750110 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1"} err="failed to get container status \"46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1\": rpc error: code = NotFound desc = could not find container \"46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1\": container with ID starting with 46b5744149c17b3e5a7c2714a738e6c228ba165190f0d399c12f2de9be5512f1 not found: ID does not exist" Jul 16 17:36:28.750155 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.750123 2576 scope.go:117] "RemoveContainer" containerID="b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2" Jul 16 17:36:28.750308 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:36:28.750293 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2\": container with ID starting with b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2 not found: ID does not exist" containerID="b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2" Jul 16 17:36:28.750345 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.750308 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2"} err="failed to get container status \"b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2\": rpc error: code = NotFound desc = could not find container \"b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2\": container with ID starting with b76d33331fac440e66b4a4c6741fd7a989bdfd855e6e2428e9ad1b638e660ff2 not found: ID does not exist" Jul 16 17:36:28.750345 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.750320 2576 scope.go:117] "RemoveContainer" containerID="2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206" Jul 16 17:36:28.750527 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:36:28.750509 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206\": container with ID starting with 2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206 not found: ID does not exist" containerID="2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206" Jul 16 17:36:28.750570 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.750535 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206"} err="failed to get container status \"2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206\": rpc error: code = NotFound desc = could not find container \"2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206\": container with ID starting with 2e98e98ab70c2d847852a4bf9bb73696f781687a89b29929ed6ef561f7122206 not found: ID does not exist" Jul 16 17:36:28.750570 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.750552 2576 scope.go:117] "RemoveContainer" containerID="08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d" Jul 16 17:36:28.750782 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:36:28.750762 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d\": container with ID starting with 08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d not found: ID does not exist" containerID="08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d" Jul 16 17:36:28.750855 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.750784 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d"} err="failed to get container status \"08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d\": rpc error: code = NotFound desc = could not find container \"08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d\": container with ID starting with 08a0655b6b5fab8a86d0ce4aa4fe652893e715b24cd6841408707b9e13b8981d not found: ID does not exist" Jul 16 17:36:28.750855 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.750798 2576 scope.go:117] "RemoveContainer" containerID="9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f" Jul 16 17:36:28.751032 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:36:28.751016 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f\": container with ID starting with 9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f not found: ID does not exist" containerID="9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f" Jul 16 17:36:28.751064 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.751037 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f"} err="failed to get container status \"9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f\": rpc error: code = NotFound desc = could not find container \"9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f\": container with ID starting with 9629d54638e603a3264f834cf5880df28b679aee108b834644c6ee7640bbf56f not found: ID does not exist" Jul 16 17:36:28.751064 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.751050 2576 scope.go:117] "RemoveContainer" containerID="0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6" Jul 16 17:36:28.751272 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:36:28.751256 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6\": container with ID starting with 0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6 not found: ID does not exist" containerID="0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6" Jul 16 17:36:28.751317 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.751276 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6"} err="failed to get container status \"0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6\": rpc error: code = NotFound desc = could not find container \"0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6\": container with ID starting with 0cc81f975c9fd67ff2b08b9eac309063e4d9ae3edc14c7a859b59d10437b0af6 not found: ID does not exist" Jul 16 17:36:28.770114 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.770084 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" path="/var/lib/kubelet/pods/02ce6193-1165-4e1e-a5a0-e11d4631f55c/volumes" Jul 16 17:36:28.783955 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.783924 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:36:28.784195 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784183 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy-web" Jul 16 17:36:28.784234 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784198 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy-web" Jul 16 17:36:28.784234 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784210 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="alertmanager" Jul 16 17:36:28.784234 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784216 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="alertmanager" Jul 16 17:36:28.784234 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784223 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy-metric" Jul 16 17:36:28.784234 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784228 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy-metric" Jul 16 17:36:28.784234 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784234 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="prom-label-proxy" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784239 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="prom-label-proxy" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784250 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784256 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784263 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="config-reloader" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784268 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="config-reloader" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784277 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="init-config-reloader" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784282 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="init-config-reloader" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784328 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="config-reloader" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784335 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy-metric" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784341 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="alertmanager" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784348 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy-web" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784353 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="kube-rbac-proxy" Jul 16 17:36:28.784406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.784357 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ce6193-1165-4e1e-a5a0-e11d4631f55c" containerName="prom-label-proxy" Jul 16 17:36:28.787686 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.787665 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.790725 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.790690 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gbl27\"" Jul 16 17:36:28.790838 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.790692 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 17:36:28.790838 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.790692 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 17:36:28.791099 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.791079 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 17:36:28.791099 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.791085 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 17:36:28.791240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.791223 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 17:36:28.791240 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.791230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 17:36:28.791329 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.791251 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 17:36:28.791459 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.791446 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 17:36:28.800483 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.800457 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 17:36:28.813491 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.813466 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:36:28.926398 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.926585 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b8c9990-6949-406b-aa0f-3caad6269a15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.926585 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926449 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.926585 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b8c9990-6949-406b-aa0f-3caad6269a15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.926585 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b8c9990-6949-406b-aa0f-3caad6269a15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.926785 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.926785 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.926785 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.926785 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.926785 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b8c9990-6949-406b-aa0f-3caad6269a15-config-out\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.927005 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b8c9990-6949-406b-aa0f-3caad6269a15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.927005 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8w8\" (UniqueName: \"kubernetes.io/projected/1b8c9990-6949-406b-aa0f-3caad6269a15-kube-api-access-nc8w8\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:28.927005 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:28.926879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-web-config\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028327 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b8c9990-6949-406b-aa0f-3caad6269a15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028327 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b8c9990-6949-406b-aa0f-3caad6269a15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028327 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028327 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b8c9990-6949-406b-aa0f-3caad6269a15-config-out\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b8c9990-6949-406b-aa0f-3caad6269a15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8w8\" (UniqueName: \"kubernetes.io/projected/1b8c9990-6949-406b-aa0f-3caad6269a15-kube-api-access-nc8w8\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-web-config\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b8c9990-6949-406b-aa0f-3caad6269a15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.028953 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.028753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b8c9990-6949-406b-aa0f-3caad6269a15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.029446 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.029375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b8c9990-6949-406b-aa0f-3caad6269a15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.030182 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.030154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b8c9990-6949-406b-aa0f-3caad6269a15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.031348 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.031319 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b8c9990-6949-406b-aa0f-3caad6269a15-config-out\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.031459 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.031396 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b8c9990-6949-406b-aa0f-3caad6269a15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.031824 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.031672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.031942 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.031918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.032038 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.032009 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.032460 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.032308 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.032632 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.032612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.032800 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.032781 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.033469 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.033445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b8c9990-6949-406b-aa0f-3caad6269a15-web-config\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.042368 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.042336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8w8\" (UniqueName: \"kubernetes.io/projected/1b8c9990-6949-406b-aa0f-3caad6269a15-kube-api-access-nc8w8\") pod \"alertmanager-main-0\" (UID: \"1b8c9990-6949-406b-aa0f-3caad6269a15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.097347 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.097313 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:29.248802 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.248771 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:36:29.251879 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:36:29.251853 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8c9990_6949_406b_aa0f_3caad6269a15.slice/crio-7597ebad3354d96922a356c92c30099158cc4ecbd15977916da4d7a32f00cd6e WatchSource:0}: Error finding container 7597ebad3354d96922a356c92c30099158cc4ecbd15977916da4d7a32f00cd6e: Status 404 returned error can't find the container with id 7597ebad3354d96922a356c92c30099158cc4ecbd15977916da4d7a32f00cd6e Jul 16 17:36:29.698406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.698368 2576 generic.go:356] "Generic (PLEG): container finished" podID="1b8c9990-6949-406b-aa0f-3caad6269a15" containerID="c40b554e7ed1bac27f6d77b34777dcadb34d3c7dafce5abd87080f5f88c2950e" exitCode=0 Jul 16 17:36:29.698547 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.698453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b8c9990-6949-406b-aa0f-3caad6269a15","Type":"ContainerDied","Data":"c40b554e7ed1bac27f6d77b34777dcadb34d3c7dafce5abd87080f5f88c2950e"} Jul 16 17:36:29.698547 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:29.698521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b8c9990-6949-406b-aa0f-3caad6269a15","Type":"ContainerStarted","Data":"7597ebad3354d96922a356c92c30099158cc4ecbd15977916da4d7a32f00cd6e"} Jul 16 17:36:30.706624 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.706586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b8c9990-6949-406b-aa0f-3caad6269a15","Type":"ContainerStarted","Data":"375bb79998de55e049c3433dfc0a86410d3862d5f74555dd3da0a4973141b388"} Jul 16 17:36:30.707099 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.706651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b8c9990-6949-406b-aa0f-3caad6269a15","Type":"ContainerStarted","Data":"3b67e8c2ac09c41b4ff221245347dcd85b5fa8c625b4c28fd8c1f2da96722f7c"} Jul 16 17:36:30.707099 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.706684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b8c9990-6949-406b-aa0f-3caad6269a15","Type":"ContainerStarted","Data":"c05636ed5c99dfa859fe1f7e18c647f87b2d1912bf7f487e8c79cd5d7d8d75cf"} Jul 16 17:36:30.707099 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.706742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b8c9990-6949-406b-aa0f-3caad6269a15","Type":"ContainerStarted","Data":"9735c14184d1655a42ab881960fd4be61d7abaee564cc27e57b157abcddfb168"} Jul 16 17:36:30.707099 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.706767 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b8c9990-6949-406b-aa0f-3caad6269a15","Type":"ContainerStarted","Data":"e747676a5c3e292e0660fa64147b19e930d47901fd1dbd1943752c4bd84fb015"} Jul 16 17:36:30.707099 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.706789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b8c9990-6949-406b-aa0f-3caad6269a15","Type":"ContainerStarted","Data":"36d3284b2db6ac3cc65ead8de3ababdf9153ad3a9ce291d766718415df5be237"} Jul 16 17:36:30.749947 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.749890 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.731415058 podStartE2EDuration="2.749874306s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.459248ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:30.748718755 +0000 UTC m=+234.410282384" watchObservedRunningTime="2026-07-16 17:36:30.749874306 +0000 UTC m=+234.411437940" Jul 16 17:36:30.887594 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.887560 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq"] Jul 16 17:36:30.891031 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.891013 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:30.895189 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.895157 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 17:36:30.895304 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.895212 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bfqvf\"" Jul 16 17:36:30.895466 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.895450 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 17:36:30.895954 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.895939 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 17:36:30.896291 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.896274 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 17:36:30.896800 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.896777 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 17:36:30.926042 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.926013 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 17:36:30.943786 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.943753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5vz\" (UniqueName: \"kubernetes.io/projected/f357bf8c-2d12-44d9-a8d3-58c914163edd-kube-api-access-vb5vz\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:30.943786 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.943787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f357bf8c-2d12-44d9-a8d3-58c914163edd-metrics-client-ca\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:30.943971 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.943817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f357bf8c-2d12-44d9-a8d3-58c914163edd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:30.943971 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.943842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-federate-client-tls\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:30.943971 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.943903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:30.943971 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.943945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f357bf8c-2d12-44d9-a8d3-58c914163edd-serving-certs-ca-bundle\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:30.944209 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.943973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-telemeter-client-tls\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:30.944209 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.943998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-secret-telemeter-client\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:30.946783 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:30.946759 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq"] Jul 16 17:36:31.044678 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.044584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5vz\" (UniqueName: \"kubernetes.io/projected/f357bf8c-2d12-44d9-a8d3-58c914163edd-kube-api-access-vb5vz\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.044678 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.044624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f357bf8c-2d12-44d9-a8d3-58c914163edd-metrics-client-ca\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.044678 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.044654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f357bf8c-2d12-44d9-a8d3-58c914163edd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.044678 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.044672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-federate-client-tls\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.045018 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.044731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.045018 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.044765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f357bf8c-2d12-44d9-a8d3-58c914163edd-serving-certs-ca-bundle\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.045018 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.044804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-telemeter-client-tls\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.045018 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.044855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-secret-telemeter-client\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.045563 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.045530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f357bf8c-2d12-44d9-a8d3-58c914163edd-metrics-client-ca\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.045670 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.045654 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f357bf8c-2d12-44d9-a8d3-58c914163edd-serving-certs-ca-bundle\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.046041 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.046014 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f357bf8c-2d12-44d9-a8d3-58c914163edd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.047548 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.047520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-secret-telemeter-client\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.047734 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.047690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-telemeter-client-tls\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.047828 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.047802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.048044 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.048022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f357bf8c-2d12-44d9-a8d3-58c914163edd-federate-client-tls\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.057161 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.057133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5vz\" (UniqueName: \"kubernetes.io/projected/f357bf8c-2d12-44d9-a8d3-58c914163edd-kube-api-access-vb5vz\") pod \"telemeter-client-54fbbb49cc-x5kfq\" (UID: \"f357bf8c-2d12-44d9-a8d3-58c914163edd\") " pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.200775 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.200740 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" Jul 16 17:36:31.367845 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.367823 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq"] Jul 16 17:36:31.369891 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:36:31.369866 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf357bf8c_2d12_44d9_a8d3_58c914163edd.slice/crio-d16669608d6ae59775de458ab3203d4b9f07b6a0e4e44cc10ebcd59487a925c7 WatchSource:0}: Error finding container d16669608d6ae59775de458ab3203d4b9f07b6a0e4e44cc10ebcd59487a925c7: Status 404 returned error can't find the container with id d16669608d6ae59775de458ab3203d4b9f07b6a0e4e44cc10ebcd59487a925c7 Jul 16 17:36:31.713062 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:31.713030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" event={"ID":"f357bf8c-2d12-44d9-a8d3-58c914163edd","Type":"ContainerStarted","Data":"d16669608d6ae59775de458ab3203d4b9f07b6a0e4e44cc10ebcd59487a925c7"} Jul 16 17:36:33.722853 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:33.722819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" event={"ID":"f357bf8c-2d12-44d9-a8d3-58c914163edd","Type":"ContainerStarted","Data":"850c248583998b03089338bdfcb9c34941e3058672d3f2ba956775b1afc3660b"} Jul 16 17:36:33.723263 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:33.722902 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" event={"ID":"f357bf8c-2d12-44d9-a8d3-58c914163edd","Type":"ContainerStarted","Data":"9af69519846723849e77f4ae6ee9f0d911baebbf183e478ddcc66adb8b7caf77"} Jul 16 17:36:33.723263 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:33.722922 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" event={"ID":"f357bf8c-2d12-44d9-a8d3-58c914163edd","Type":"ContainerStarted","Data":"8f356a03d1a563081d760ca66d5bccd13509bd371a88900a23a72dfdf7be136e"} Jul 16 17:36:33.751255 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:36:33.751202 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54fbbb49cc-x5kfq" podStartSLOduration=1.9899131799999998 podStartE2EDuration="3.751186895s" totalImagesPullingTime="1.761273715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:33.750358523 +0000 UTC m=+237.411922167" watchObservedRunningTime="2026-07-16 17:36:33.751186895 +0000 UTC m=+237.412750527" Jul 16 17:37:36.644219 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:37:36.644189 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:37:36.651036 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:37:36.651016 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 17:37:36.652276 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:37:36.652252 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:38:26.612855 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.612821 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx"] Jul 16 17:38:26.615921 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.615899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" Jul 16 17:38:26.616189 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.616163 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd"] Jul 16 17:38:26.618542 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.618519 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 17:38:26.619294 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.619274 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.619509 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.619477 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 16 17:38:26.619509 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.619483 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 17:38:26.619672 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.619592 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xzdw8\"" Jul 16 17:38:26.619755 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.619673 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 17:38:26.623182 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.622908 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 16 17:38:26.623680 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.623664 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 16 17:38:26.623856 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.623838 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 16 17:38:26.624923 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.624906 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 16 17:38:26.628871 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.628851 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx"] Jul 16 17:38:26.632303 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.632282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd"] Jul 16 17:38:26.799435 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.799398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.799642 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.799455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-hub\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.799642 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.799529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db0a7c68-6fb9-4520-92a6-5d587ae94214-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56b99c674d-jm8hx\" (UID: \"db0a7c68-6fb9-4520-92a6-5d587ae94214\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" Jul 16 17:38:26.799642 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.799616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.799860 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.799649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mxtf\" (UniqueName: \"kubernetes.io/projected/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-kube-api-access-8mxtf\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.799860 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.799681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.799860 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.799767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99gm\" (UniqueName: \"kubernetes.io/projected/db0a7c68-6fb9-4520-92a6-5d587ae94214-kube-api-access-j99gm\") pod \"managed-serviceaccount-addon-agent-56b99c674d-jm8hx\" (UID: \"db0a7c68-6fb9-4520-92a6-5d587ae94214\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" Jul 16 17:38:26.799860 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.799791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-ca\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.900919 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.900882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mxtf\" (UniqueName: \"kubernetes.io/projected/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-kube-api-access-8mxtf\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.901116 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.900926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.901116 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.900969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99gm\" (UniqueName: \"kubernetes.io/projected/db0a7c68-6fb9-4520-92a6-5d587ae94214-kube-api-access-j99gm\") pod \"managed-serviceaccount-addon-agent-56b99c674d-jm8hx\" (UID: \"db0a7c68-6fb9-4520-92a6-5d587ae94214\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" Jul 16 17:38:26.901116 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.900992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-ca\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.901116 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.901025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.901116 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.901062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-hub\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.901360 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.901205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db0a7c68-6fb9-4520-92a6-5d587ae94214-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56b99c674d-jm8hx\" (UID: \"db0a7c68-6fb9-4520-92a6-5d587ae94214\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" Jul 16 17:38:26.901360 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.901307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.901917 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.901890 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.903979 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.903948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db0a7c68-6fb9-4520-92a6-5d587ae94214-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56b99c674d-jm8hx\" (UID: \"db0a7c68-6fb9-4520-92a6-5d587ae94214\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" Jul 16 17:38:26.904071 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.904008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.904071 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.904043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-hub\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.904071 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.904064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-ca\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.904228 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.904207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.910283 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.910255 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mxtf\" (UniqueName: \"kubernetes.io/projected/d9ff6b76-812a-49bc-b2c2-e6619ebcac49-kube-api-access-8mxtf\") pod \"cluster-proxy-proxy-agent-66b87b88b6-6ldfd\" (UID: \"d9ff6b76-812a-49bc-b2c2-e6619ebcac49\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:26.910658 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.910633 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99gm\" (UniqueName: \"kubernetes.io/projected/db0a7c68-6fb9-4520-92a6-5d587ae94214-kube-api-access-j99gm\") pod \"managed-serviceaccount-addon-agent-56b99c674d-jm8hx\" (UID: \"db0a7c68-6fb9-4520-92a6-5d587ae94214\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" Jul 16 17:38:26.936105 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.936067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" Jul 16 17:38:26.944276 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:26.944247 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" Jul 16 17:38:27.093566 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:27.093541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd"] Jul 16 17:38:27.096282 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:38:27.096258 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9ff6b76_812a_49bc_b2c2_e6619ebcac49.slice/crio-8294db639b8c3602bfa9a03b42b8b0fdd12cf59cd6f428d63cfd15ea43f4f36a WatchSource:0}: Error finding container 8294db639b8c3602bfa9a03b42b8b0fdd12cf59cd6f428d63cfd15ea43f4f36a: Status 404 returned error can't find the container with id 8294db639b8c3602bfa9a03b42b8b0fdd12cf59cd6f428d63cfd15ea43f4f36a Jul 16 17:38:27.098060 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:27.098046 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:38:27.102042 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:27.101977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx"] Jul 16 17:38:27.104310 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:38:27.104287 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb0a7c68_6fb9_4520_92a6_5d587ae94214.slice/crio-0a232199526506c13290e40ae0d3d01b3959997daa50a938f260e72f6844dcb1 WatchSource:0}: Error finding container 0a232199526506c13290e40ae0d3d01b3959997daa50a938f260e72f6844dcb1: Status 404 returned error can't find the container with id 0a232199526506c13290e40ae0d3d01b3959997daa50a938f260e72f6844dcb1 Jul 16 17:38:27.146543 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:27.146505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" event={"ID":"d9ff6b76-812a-49bc-b2c2-e6619ebcac49","Type":"ContainerStarted","Data":"8294db639b8c3602bfa9a03b42b8b0fdd12cf59cd6f428d63cfd15ea43f4f36a"} Jul 16 17:38:27.147626 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:27.147603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" event={"ID":"db0a7c68-6fb9-4520-92a6-5d587ae94214","Type":"ContainerStarted","Data":"0a232199526506c13290e40ae0d3d01b3959997daa50a938f260e72f6844dcb1"} Jul 16 17:38:30.164448 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:30.164122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" event={"ID":"db0a7c68-6fb9-4520-92a6-5d587ae94214","Type":"ContainerStarted","Data":"efdc89613a372cc2e5c5d9c82c2417d81ecf06e3f21c7a8a8b3854bcdee622fd"} Jul 16 17:38:30.182836 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:30.182782 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b99c674d-jm8hx" podStartSLOduration=1.280313944 podStartE2EDuration="4.182763587s" totalImagesPullingTime="2.902449643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:30.181278023 +0000 UTC m=+353.842841658" watchObservedRunningTime="2026-07-16 17:38:30.182763587 +0000 UTC m=+353.844327219" Jul 16 17:38:32.173505 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:32.173467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" event={"ID":"d9ff6b76-812a-49bc-b2c2-e6619ebcac49","Type":"ContainerStarted","Data":"5de46dc75db7bd6e5c2b47ad5c43279d6f051fcdc6ba4bdf7e26c6f404195110"} Jul 16 17:38:32.173917 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:32.173514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" event={"ID":"d9ff6b76-812a-49bc-b2c2-e6619ebcac49","Type":"ContainerStarted","Data":"ac8c98e193b93828817bddfd02478b9865bde9876a0b6d9fd008267b66823a49"} Jul 16 17:38:32.173917 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:32.173527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" event={"ID":"d9ff6b76-812a-49bc-b2c2-e6619ebcac49","Type":"ContainerStarted","Data":"7127879ea960088f72d5d6e301600d717e0a0c4f00f04e4960e7241d86084e89"} Jul 16 17:38:32.197977 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:38:32.197924 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66b87b88b6-6ldfd" podStartSLOduration=1.863237475 podStartE2EDuration="6.197908492s" totalImagesPullingTime="4.334671017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:32.196604652 +0000 UTC m=+355.858168285" watchObservedRunningTime="2026-07-16 17:38:32.197908492 +0000 UTC m=+355.859472160" Jul 16 17:40:52.612804 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.612754 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm"] Jul 16 17:40:52.616038 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.616022 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.618745 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.618724 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 17:40:52.618879 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.618790 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 17:40:52.619652 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.619630 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zlpzt\"" Jul 16 17:40:52.619788 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.619676 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 17:40:52.619788 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.619638 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:40:52.619957 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.619941 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 17:40:52.631164 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.631141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm"] Jul 16 17:40:52.753188 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.753153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/857205c0-47c2-43e4-90ce-d45ff2779866-cert\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.753389 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.753203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/857205c0-47c2-43e4-90ce-d45ff2779866-metrics-cert\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.753389 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.753289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/857205c0-47c2-43e4-90ce-d45ff2779866-manager-config\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.753389 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.753354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktj5\" (UniqueName: \"kubernetes.io/projected/857205c0-47c2-43e4-90ce-d45ff2779866-kube-api-access-xktj5\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.854542 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.854501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktj5\" (UniqueName: \"kubernetes.io/projected/857205c0-47c2-43e4-90ce-d45ff2779866-kube-api-access-xktj5\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.854753 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.854561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/857205c0-47c2-43e4-90ce-d45ff2779866-cert\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.854753 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.854606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/857205c0-47c2-43e4-90ce-d45ff2779866-metrics-cert\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.854753 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.854644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/857205c0-47c2-43e4-90ce-d45ff2779866-manager-config\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.855371 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.855349 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/857205c0-47c2-43e4-90ce-d45ff2779866-manager-config\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.857196 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.857173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/857205c0-47c2-43e4-90ce-d45ff2779866-cert\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.857292 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.857231 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/857205c0-47c2-43e4-90ce-d45ff2779866-metrics-cert\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.864483 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.864422 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktj5\" (UniqueName: \"kubernetes.io/projected/857205c0-47c2-43e4-90ce-d45ff2779866-kube-api-access-xktj5\") pod \"lws-controller-manager-65794b4fd8-5kkmm\" (UID: \"857205c0-47c2-43e4-90ce-d45ff2779866\") " pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:52.925431 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:52.925388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:53.076489 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:53.076465 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm"] Jul 16 17:40:53.078933 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:40:53.078908 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857205c0_47c2_43e4_90ce_d45ff2779866.slice/crio-9f91447d4351a47ff6299f5f3fb5d6cae2600f161e723bdd2bc6d79c69cc98ef WatchSource:0}: Error finding container 9f91447d4351a47ff6299f5f3fb5d6cae2600f161e723bdd2bc6d79c69cc98ef: Status 404 returned error can't find the container with id 9f91447d4351a47ff6299f5f3fb5d6cae2600f161e723bdd2bc6d79c69cc98ef Jul 16 17:40:53.740873 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:53.740833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" event={"ID":"857205c0-47c2-43e4-90ce-d45ff2779866","Type":"ContainerStarted","Data":"9f91447d4351a47ff6299f5f3fb5d6cae2600f161e723bdd2bc6d79c69cc98ef"} Jul 16 17:40:56.756076 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:56.756042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" event={"ID":"857205c0-47c2-43e4-90ce-d45ff2779866","Type":"ContainerStarted","Data":"b680d8e43407913696a96e482b3e228fa7470266b99ad6363c6549b58126469d"} Jul 16 17:40:56.756468 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:56.756179 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:40:56.774980 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:40:56.774933 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" podStartSLOduration=1.769414291 podStartE2EDuration="4.774918535s" totalImagesPullingTime="3.005504244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:40:56.773935082 +0000 UTC m=+500.435498714" watchObservedRunningTime="2026-07-16 17:40:56.774918535 +0000 UTC m=+500.436482167" Jul 16 17:41:00.804294 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.804262 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4"] Jul 16 17:41:00.807759 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.807741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.810996 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.810955 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 17:41:00.810996 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.810984 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 17:41:00.811152 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.811070 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n44d8\"" Jul 16 17:41:00.811506 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.811487 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 17:41:00.813135 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.813120 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 17:41:00.825712 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.825654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0f7a6884-e166-445a-ac18-6fcb34814859-apiservice-cert\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.825855 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.825747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f7a6884-e166-445a-ac18-6fcb34814859-tmp\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.825855 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.825803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0f7a6884-e166-445a-ac18-6fcb34814859-manifests\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.825855 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.825836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5st2f\" (UniqueName: \"kubernetes.io/projected/0f7a6884-e166-445a-ac18-6fcb34814859-kube-api-access-5st2f\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.826053 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.825863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f7a6884-e166-445a-ac18-6fcb34814859-webhook-cert\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.827226 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.827183 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4"] Jul 16 17:41:00.926937 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.926899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0f7a6884-e166-445a-ac18-6fcb34814859-apiservice-cert\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.927126 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.926946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f7a6884-e166-445a-ac18-6fcb34814859-tmp\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.927126 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.926979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0f7a6884-e166-445a-ac18-6fcb34814859-manifests\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.927126 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.927005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st2f\" (UniqueName: \"kubernetes.io/projected/0f7a6884-e166-445a-ac18-6fcb34814859-kube-api-access-5st2f\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.927126 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.927023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f7a6884-e166-445a-ac18-6fcb34814859-webhook-cert\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.927471 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.927449 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0f7a6884-e166-445a-ac18-6fcb34814859-manifests\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.929246 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.929220 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f7a6884-e166-445a-ac18-6fcb34814859-tmp\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.929361 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.929341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f7a6884-e166-445a-ac18-6fcb34814859-webhook-cert\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.929536 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.929518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0f7a6884-e166-445a-ac18-6fcb34814859-apiservice-cert\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:00.939161 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:00.939137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st2f\" (UniqueName: \"kubernetes.io/projected/0f7a6884-e166-445a-ac18-6fcb34814859-kube-api-access-5st2f\") pod \"opendatahub-operator-controller-manager-674ff8dd58-jf9t4\" (UID: \"0f7a6884-e166-445a-ac18-6fcb34814859\") " pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:01.117557 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:01.117475 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:01.261841 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:01.261800 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4"] Jul 16 17:41:01.264944 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:41:01.264920 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7a6884_e166_445a_ac18_6fcb34814859.slice/crio-43fbb97d32af9886fada28903b2ca1f74bac4efdb85146d4119547f5ee9dbb18 WatchSource:0}: Error finding container 43fbb97d32af9886fada28903b2ca1f74bac4efdb85146d4119547f5ee9dbb18: Status 404 returned error can't find the container with id 43fbb97d32af9886fada28903b2ca1f74bac4efdb85146d4119547f5ee9dbb18 Jul 16 17:41:01.778404 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:01.778372 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" event={"ID":"0f7a6884-e166-445a-ac18-6fcb34814859","Type":"ContainerStarted","Data":"43fbb97d32af9886fada28903b2ca1f74bac4efdb85146d4119547f5ee9dbb18"} Jul 16 17:41:04.798848 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:04.798801 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" event={"ID":"0f7a6884-e166-445a-ac18-6fcb34814859","Type":"ContainerStarted","Data":"9c5c48d6c84778397f7b75bb9284c496d41903d77e392f952eede62e47431cab"} Jul 16 17:41:06.809435 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:06.809401 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f7a6884-e166-445a-ac18-6fcb34814859" containerID="9c5c48d6c84778397f7b75bb9284c496d41903d77e392f952eede62e47431cab" exitCode=0 Jul 16 17:41:06.809913 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:06.809431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" event={"ID":"0f7a6884-e166-445a-ac18-6fcb34814859","Type":"ContainerDied","Data":"9c5c48d6c84778397f7b75bb9284c496d41903d77e392f952eede62e47431cab"} Jul 16 17:41:07.762724 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:07.762678 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65794b4fd8-5kkmm" Jul 16 17:41:07.817091 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:07.817056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" event={"ID":"0f7a6884-e166-445a-ac18-6fcb34814859","Type":"ContainerStarted","Data":"34536f83ae74d3a8e128b8566632318d43b9a6acf03bf3673c3fd2baeabfb62c"} Jul 16 17:41:07.817437 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:07.817279 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:07.843478 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:07.843372 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" podStartSLOduration=2.605304334 podStartE2EDuration="7.843353225s" totalImagesPullingTime="4.204269343s" totalInitContainerRuntime="1.033779548s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:07.842886878 +0000 UTC m=+511.504450524" watchObservedRunningTime="2026-07-16 17:41:07.843353225 +0000 UTC m=+511.504916859" Jul 16 17:41:18.824098 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:18.824066 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-674ff8dd58-jf9t4" Jul 16 17:41:21.125205 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.125168 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc"] Jul 16 17:41:21.270780 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.270688 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc"] Jul 16 17:41:21.270954 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.270806 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.275451 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.275426 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 17:41:21.275575 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.275503 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 17:41:21.278862 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.278842 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 17:41:21.278862 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.278844 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 17:41:21.279008 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.278969 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ztjdc\"" Jul 16 17:41:21.398742 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.398688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69be8998-c243-473c-a48e-26b5f9a5d0bb-tmp\") pod \"kube-auth-proxy-5966557cf8-xn9sc\" (UID: \"69be8998-c243-473c-a48e-26b5f9a5d0bb\") " pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.399044 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.399008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69be8998-c243-473c-a48e-26b5f9a5d0bb-tls-certs\") pod \"kube-auth-proxy-5966557cf8-xn9sc\" (UID: \"69be8998-c243-473c-a48e-26b5f9a5d0bb\") " pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.399189 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.399068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fzhf\" (UniqueName: \"kubernetes.io/projected/69be8998-c243-473c-a48e-26b5f9a5d0bb-kube-api-access-7fzhf\") pod \"kube-auth-proxy-5966557cf8-xn9sc\" (UID: \"69be8998-c243-473c-a48e-26b5f9a5d0bb\") " pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.499684 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.499648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69be8998-c243-473c-a48e-26b5f9a5d0bb-tmp\") pod \"kube-auth-proxy-5966557cf8-xn9sc\" (UID: \"69be8998-c243-473c-a48e-26b5f9a5d0bb\") " pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.499929 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.499749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69be8998-c243-473c-a48e-26b5f9a5d0bb-tls-certs\") pod \"kube-auth-proxy-5966557cf8-xn9sc\" (UID: \"69be8998-c243-473c-a48e-26b5f9a5d0bb\") " pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.499929 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.499788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fzhf\" (UniqueName: \"kubernetes.io/projected/69be8998-c243-473c-a48e-26b5f9a5d0bb-kube-api-access-7fzhf\") pod \"kube-auth-proxy-5966557cf8-xn9sc\" (UID: \"69be8998-c243-473c-a48e-26b5f9a5d0bb\") " pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.501998 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.501971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69be8998-c243-473c-a48e-26b5f9a5d0bb-tmp\") pod \"kube-auth-proxy-5966557cf8-xn9sc\" (UID: \"69be8998-c243-473c-a48e-26b5f9a5d0bb\") " pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.502195 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.502178 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69be8998-c243-473c-a48e-26b5f9a5d0bb-tls-certs\") pod \"kube-auth-proxy-5966557cf8-xn9sc\" (UID: \"69be8998-c243-473c-a48e-26b5f9a5d0bb\") " pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.512639 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.512609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fzhf\" (UniqueName: \"kubernetes.io/projected/69be8998-c243-473c-a48e-26b5f9a5d0bb-kube-api-access-7fzhf\") pod \"kube-auth-proxy-5966557cf8-xn9sc\" (UID: \"69be8998-c243-473c-a48e-26b5f9a5d0bb\") " pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.580266 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.580234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" Jul 16 17:41:21.718574 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.718541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc"] Jul 16 17:41:21.720452 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:41:21.720420 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69be8998_c243_473c_a48e_26b5f9a5d0bb.slice/crio-29b30c9725f174a7d25e1d1d0f4eef4c2ef4a76b2a844b5850e306ee3efd7eb1 WatchSource:0}: Error finding container 29b30c9725f174a7d25e1d1d0f4eef4c2ef4a76b2a844b5850e306ee3efd7eb1: Status 404 returned error can't find the container with id 29b30c9725f174a7d25e1d1d0f4eef4c2ef4a76b2a844b5850e306ee3efd7eb1 Jul 16 17:41:21.882480 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:21.882438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" event={"ID":"69be8998-c243-473c-a48e-26b5f9a5d0bb","Type":"ContainerStarted","Data":"29b30c9725f174a7d25e1d1d0f4eef4c2ef4a76b2a844b5850e306ee3efd7eb1"} Jul 16 17:41:25.901812 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:25.901773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" event={"ID":"69be8998-c243-473c-a48e-26b5f9a5d0bb","Type":"ContainerStarted","Data":"90a661c4c1faeb66c16a82ffd315a7a60c990d1826070bb73fc93ccb31d5fbeb"} Jul 16 17:41:25.922784 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:41:25.922732 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5966557cf8-xn9sc" podStartSLOduration=1.688445966 podStartE2EDuration="4.922714862s" totalImagesPullingTime="3.234268896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:25.921142211 +0000 UTC m=+529.582705856" watchObservedRunningTime="2026-07-16 17:41:25.922714862 +0000 UTC m=+529.584278485" Jul 16 17:42:36.673815 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:42:36.673785 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:42:36.682050 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:42:36.682024 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:43:59.888874 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.888842 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:43:59.892167 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.892150 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:43:59.895001 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.894975 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 17:43:59.895112 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.895038 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 17:43:59.895112 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.894983 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rbdx7\"" Jul 16 17:43:59.895576 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.895563 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 17:43:59.907439 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.907415 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:43:59.958009 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.957976 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:43:59.987103 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.987066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7k4\" (UniqueName: \"kubernetes.io/projected/312b41b6-d2e8-4b5b-910f-16eff440ff58-kube-api-access-gr7k4\") pod \"limitador-limitador-54d6d988d5-g662c\" (UID: \"312b41b6-d2e8-4b5b-910f-16eff440ff58\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:43:59.987288 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:43:59.987198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/312b41b6-d2e8-4b5b-910f-16eff440ff58-config-file\") pod \"limitador-limitador-54d6d988d5-g662c\" (UID: \"312b41b6-d2e8-4b5b-910f-16eff440ff58\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:44:00.088646 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.088611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/312b41b6-d2e8-4b5b-910f-16eff440ff58-config-file\") pod \"limitador-limitador-54d6d988d5-g662c\" (UID: \"312b41b6-d2e8-4b5b-910f-16eff440ff58\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:44:00.088840 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.088666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7k4\" (UniqueName: \"kubernetes.io/projected/312b41b6-d2e8-4b5b-910f-16eff440ff58-kube-api-access-gr7k4\") pod \"limitador-limitador-54d6d988d5-g662c\" (UID: \"312b41b6-d2e8-4b5b-910f-16eff440ff58\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:44:00.089380 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.089357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/312b41b6-d2e8-4b5b-910f-16eff440ff58-config-file\") pod \"limitador-limitador-54d6d988d5-g662c\" (UID: \"312b41b6-d2e8-4b5b-910f-16eff440ff58\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:44:00.103640 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.103616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7k4\" (UniqueName: \"kubernetes.io/projected/312b41b6-d2e8-4b5b-910f-16eff440ff58-kube-api-access-gr7k4\") pod \"limitador-limitador-54d6d988d5-g662c\" (UID: \"312b41b6-d2e8-4b5b-910f-16eff440ff58\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:44:00.202948 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.202914 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:44:00.345301 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.345267 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:44:00.349022 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:44:00.348993 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312b41b6_d2e8_4b5b_910f_16eff440ff58.slice/crio-7e78ba265c94cc2edab572f0fbc6f669f52df4dc7c8fea2900967eb820cffa21 WatchSource:0}: Error finding container 7e78ba265c94cc2edab572f0fbc6f669f52df4dc7c8fea2900967eb820cffa21: Status 404 returned error can't find the container with id 7e78ba265c94cc2edab572f0fbc6f669f52df4dc7c8fea2900967eb820cffa21 Jul 16 17:44:00.350810 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.350792 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:44:00.585058 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.584974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" event={"ID":"312b41b6-d2e8-4b5b-910f-16eff440ff58","Type":"ContainerStarted","Data":"7e78ba265c94cc2edab572f0fbc6f669f52df4dc7c8fea2900967eb820cffa21"} Jul 16 17:44:00.614735 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.614687 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-twfc8"] Jul 16 17:44:00.619008 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.618989 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-twfc8" Jul 16 17:44:00.621201 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.621179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-95q7m\"" Jul 16 17:44:00.624921 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.624899 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-twfc8"] Jul 16 17:44:00.695331 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.695286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxj5\" (UniqueName: \"kubernetes.io/projected/6a38dd89-08e8-474c-9eee-7ec11ad195c1-kube-api-access-tmxj5\") pod \"authorino-75d556bd77-twfc8\" (UID: \"6a38dd89-08e8-474c-9eee-7ec11ad195c1\") " pod="kuadrant-system/authorino-75d556bd77-twfc8" Jul 16 17:44:00.796024 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.795987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxj5\" (UniqueName: \"kubernetes.io/projected/6a38dd89-08e8-474c-9eee-7ec11ad195c1-kube-api-access-tmxj5\") pod \"authorino-75d556bd77-twfc8\" (UID: \"6a38dd89-08e8-474c-9eee-7ec11ad195c1\") " pod="kuadrant-system/authorino-75d556bd77-twfc8" Jul 16 17:44:00.805534 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.805507 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxj5\" (UniqueName: \"kubernetes.io/projected/6a38dd89-08e8-474c-9eee-7ec11ad195c1-kube-api-access-tmxj5\") pod \"authorino-75d556bd77-twfc8\" (UID: \"6a38dd89-08e8-474c-9eee-7ec11ad195c1\") " pod="kuadrant-system/authorino-75d556bd77-twfc8" Jul 16 17:44:00.929449 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:00.929407 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-twfc8" Jul 16 17:44:01.105551 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:01.105492 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-twfc8"] Jul 16 17:44:01.593002 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:01.592955 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-twfc8" event={"ID":"6a38dd89-08e8-474c-9eee-7ec11ad195c1","Type":"ContainerStarted","Data":"642a9a233286b8c234f5a74fa5a739b11a0f42fd5244d32949354044bcd6d336"} Jul 16 17:44:05.613738 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:05.613693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" event={"ID":"312b41b6-d2e8-4b5b-910f-16eff440ff58","Type":"ContainerStarted","Data":"1a33c458be011c053bd695c3490019ff304b56ae58064ac89814bf0310d20461"} Jul 16 17:44:05.614162 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:05.613766 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:44:05.615161 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:05.615141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-twfc8" event={"ID":"6a38dd89-08e8-474c-9eee-7ec11ad195c1","Type":"ContainerStarted","Data":"4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44"} Jul 16 17:44:05.635236 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:05.635178 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" podStartSLOduration=2.059060733 podStartE2EDuration="6.635158501s" totalImagesPullingTime="4.576097768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:44:05.633740154 +0000 UTC m=+689.295303788" watchObservedRunningTime="2026-07-16 17:44:05.635158501 +0000 UTC m=+689.296722137" Jul 16 17:44:05.651626 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:05.651579 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-twfc8" podStartSLOduration=1.836103288 podStartE2EDuration="5.651563953s" totalImagesPullingTime="3.815460665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:44:05.649748156 +0000 UTC m=+689.311311780" watchObservedRunningTime="2026-07-16 17:44:05.651563953 +0000 UTC m=+689.313127585" Jul 16 17:44:16.620821 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:16.620735 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g662c" Jul 16 17:44:36.063464 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.063384 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-twfc8"] Jul 16 17:44:36.063975 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.063562 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-twfc8" podUID="6a38dd89-08e8-474c-9eee-7ec11ad195c1" containerName="authorino" containerID="cri-o://4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44" gracePeriod=30 Jul 16 17:44:36.322429 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.322370 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-twfc8" Jul 16 17:44:36.418339 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.418303 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmxj5\" (UniqueName: \"kubernetes.io/projected/6a38dd89-08e8-474c-9eee-7ec11ad195c1-kube-api-access-tmxj5\") pod \"6a38dd89-08e8-474c-9eee-7ec11ad195c1\" (UID: \"6a38dd89-08e8-474c-9eee-7ec11ad195c1\") " Jul 16 17:44:36.420492 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.420450 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a38dd89-08e8-474c-9eee-7ec11ad195c1-kube-api-access-tmxj5" (OuterVolumeSpecName: "kube-api-access-tmxj5") pod "6a38dd89-08e8-474c-9eee-7ec11ad195c1" (UID: "6a38dd89-08e8-474c-9eee-7ec11ad195c1"). InnerVolumeSpecName "kube-api-access-tmxj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:44:36.519621 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.519588 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmxj5\" (UniqueName: \"kubernetes.io/projected/6a38dd89-08e8-474c-9eee-7ec11ad195c1-kube-api-access-tmxj5\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 17:44:36.754233 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.754199 2576 generic.go:356] "Generic (PLEG): container finished" podID="6a38dd89-08e8-474c-9eee-7ec11ad195c1" containerID="4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44" exitCode=0 Jul 16 17:44:36.754408 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.754246 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-twfc8" event={"ID":"6a38dd89-08e8-474c-9eee-7ec11ad195c1","Type":"ContainerDied","Data":"4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44"} Jul 16 17:44:36.754408 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.754261 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-twfc8" Jul 16 17:44:36.754408 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.754277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-twfc8" event={"ID":"6a38dd89-08e8-474c-9eee-7ec11ad195c1","Type":"ContainerDied","Data":"642a9a233286b8c234f5a74fa5a739b11a0f42fd5244d32949354044bcd6d336"} Jul 16 17:44:36.754408 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.754294 2576 scope.go:117] "RemoveContainer" containerID="4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44" Jul 16 17:44:36.763386 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.763367 2576 scope.go:117] "RemoveContainer" containerID="4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44" Jul 16 17:44:36.763657 ip-10-0-128-217 kubenswrapper[2576]: E0716 17:44:36.763638 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44\": container with ID starting with 4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44 not found: ID does not exist" containerID="4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44" Jul 16 17:44:36.763715 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.763667 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44"} err="failed to get container status \"4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44\": rpc error: code = NotFound desc = could not find container \"4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44\": container with ID starting with 4beccd0839d4d03cd80210c19f398a0c5f417a8030cd3f5d6d016b6cfb4ade44 not found: ID does not exist" Jul 16 17:44:36.780311 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.780277 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-twfc8"] Jul 16 17:44:36.792946 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:36.792914 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-twfc8"] Jul 16 17:44:38.770777 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:44:38.770744 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a38dd89-08e8-474c-9eee-7ec11ad195c1" path="/var/lib/kubelet/pods/6a38dd89-08e8-474c-9eee-7ec11ad195c1/volumes" Jul 16 17:45:22.119406 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:22.119364 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:45:37.907869 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:37.907835 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:45:54.285558 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.285525 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m"] Jul 16 17:45:54.286017 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.285917 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a38dd89-08e8-474c-9eee-7ec11ad195c1" containerName="authorino" Jul 16 17:45:54.286017 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.285930 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a38dd89-08e8-474c-9eee-7ec11ad195c1" containerName="authorino" Jul 16 17:45:54.286017 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.285978 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a38dd89-08e8-474c-9eee-7ec11ad195c1" containerName="authorino" Jul 16 17:45:54.290051 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.290035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.292865 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.292839 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 17:45:54.292980 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.292884 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 17:45:54.293896 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.293855 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 16 17:45:54.293989 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.293900 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kw8bd\"" Jul 16 17:45:54.299055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.299035 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m"] Jul 16 17:45:54.475941 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.475903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.475941 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.475941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898ea9e4-e232-49e6-acae-0f4dd39ca165-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.476161 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.475963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.476161 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.475980 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.476161 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.476029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.476161 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.476121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvl2t\" (UniqueName: \"kubernetes.io/projected/898ea9e4-e232-49e6-acae-0f4dd39ca165-kube-api-access-xvl2t\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.576721 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.576616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.576721 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.576649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898ea9e4-e232-49e6-acae-0f4dd39ca165-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.576721 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.576676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.576721 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.576692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.576993 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.576736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.576993 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.576793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvl2t\" (UniqueName: \"kubernetes.io/projected/898ea9e4-e232-49e6-acae-0f4dd39ca165-kube-api-access-xvl2t\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.577126 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.577098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.577258 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.577115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.577258 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.577173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.578992 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.578971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898ea9e4-e232-49e6-acae-0f4dd39ca165-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.579251 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.579232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898ea9e4-e232-49e6-acae-0f4dd39ca165-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.587147 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.587124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvl2t\" (UniqueName: \"kubernetes.io/projected/898ea9e4-e232-49e6-acae-0f4dd39ca165-kube-api-access-xvl2t\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wrw7m\" (UID: \"898ea9e4-e232-49e6-acae-0f4dd39ca165\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.601920 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.601900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:45:54.738235 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:54.738211 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m"] Jul 16 17:45:54.740848 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:45:54.740805 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898ea9e4_e232_49e6_acae_0f4dd39ca165.slice/crio-b53f9b5de7641ae9acf71ae53da28283aad3f8dadf175a4c47ded93c80d3f0a7 WatchSource:0}: Error finding container b53f9b5de7641ae9acf71ae53da28283aad3f8dadf175a4c47ded93c80d3f0a7: Status 404 returned error can't find the container with id b53f9b5de7641ae9acf71ae53da28283aad3f8dadf175a4c47ded93c80d3f0a7 Jul 16 17:45:55.116205 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:55.116164 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" event={"ID":"898ea9e4-e232-49e6-acae-0f4dd39ca165","Type":"ContainerStarted","Data":"b53f9b5de7641ae9acf71ae53da28283aad3f8dadf175a4c47ded93c80d3f0a7"} Jul 16 17:45:55.206834 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:55.206739 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:45:56.122903 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:45:56.122810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" event={"ID":"898ea9e4-e232-49e6-acae-0f4dd39ca165","Type":"ContainerStarted","Data":"c7a5c09e1bd40fe695ede0e4e1300fc9e0e3db244187ad51fb864ddc433f52c9"} Jul 16 17:46:04.603032 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:04.602937 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:46:04.603032 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:04.603008 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:46:04.615659 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:04.615631 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:46:04.642871 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:04.642805 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" podStartSLOduration=9.638952003 podStartE2EDuration="10.642785804s" totalImagesPullingTime="1.003833801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:56.156660203 +0000 UTC m=+799.818223837" watchObservedRunningTime="2026-07-16 17:46:04.642785804 +0000 UTC m=+808.304349438" Jul 16 17:46:05.173716 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:05.173674 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wrw7m" Jul 16 17:46:21.701655 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:21.701616 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:46:29.175654 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.175618 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg"] Jul 16 17:46:29.180785 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.180762 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.183419 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.183389 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 16 17:46:29.193894 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.193869 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg"] Jul 16 17:46:29.276055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.276015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.276055 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.276054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.276253 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.276078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.276253 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.276187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.276253 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.276213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.276352 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.276280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5h2c\" (UniqueName: \"kubernetes.io/projected/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-kube-api-access-r5h2c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.377400 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.377360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.377530 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.377401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.377530 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.377462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5h2c\" (UniqueName: \"kubernetes.io/projected/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-kube-api-access-r5h2c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.377530 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.377500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.377530 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.377519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.377762 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.377551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.377913 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.377869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.378035 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.377951 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.378035 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.378012 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.379841 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.379819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.380090 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.380072 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.388010 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.387979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5h2c\" (UniqueName: \"kubernetes.io/projected/d3660f9e-fd0e-4f0d-840a-9006fcb578c2-kube-api-access-r5h2c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg\" (UID: \"d3660f9e-fd0e-4f0d-840a-9006fcb578c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.490998 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.490908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:29.626815 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:29.626791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg"] Jul 16 17:46:29.629358 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:46:29.629335 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3660f9e_fd0e_4f0d_840a_9006fcb578c2.slice/crio-8b2bd0bb5666e3c4f42b03bb92a8f03427eb5311d67daa6db18eab8bd8875e17 WatchSource:0}: Error finding container 8b2bd0bb5666e3c4f42b03bb92a8f03427eb5311d67daa6db18eab8bd8875e17: Status 404 returned error can't find the container with id 8b2bd0bb5666e3c4f42b03bb92a8f03427eb5311d67daa6db18eab8bd8875e17 Jul 16 17:46:30.273911 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:30.273868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" event={"ID":"d3660f9e-fd0e-4f0d-840a-9006fcb578c2","Type":"ContainerStarted","Data":"83c8c0c55f074e1022991cf09a9d71cd2adac26a27d66c96561abbab354c7cd9"} Jul 16 17:46:30.274280 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:30.273936 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" event={"ID":"d3660f9e-fd0e-4f0d-840a-9006fcb578c2","Type":"ContainerStarted","Data":"8b2bd0bb5666e3c4f42b03bb92a8f03427eb5311d67daa6db18eab8bd8875e17"} Jul 16 17:46:30.298782 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:30.298722 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" podStartSLOduration=1.081626884 podStartE2EDuration="1.298678979s" totalImagesPullingTime="217.052095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:30.297675215 +0000 UTC m=+833.959238847" watchObservedRunningTime="2026-07-16 17:46:30.298678979 +0000 UTC m=+833.960242612" Jul 16 17:46:30.798323 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:30.798287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:46:39.491205 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:39.491171 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:39.491619 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:39.491237 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:39.503686 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:39.503661 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:40.332251 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:40.332224 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg" Jul 16 17:46:41.413109 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.413076 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n"] Jul 16 17:46:41.416498 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.416481 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.419011 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.418991 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 16 17:46:41.434611 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.434556 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n"] Jul 16 17:46:41.582305 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.582267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.582473 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.582315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2863783a-f9cd-4381-9865-e697d3d85bca-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.582473 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.582349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.582473 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.582373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmxd\" (UniqueName: \"kubernetes.io/projected/2863783a-f9cd-4381-9865-e697d3d85bca-kube-api-access-ljmxd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.582473 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.582401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.582473 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.582441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.683911 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.683837 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.683911 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.683871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.684103 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.683926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.684103 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.683960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2863783a-f9cd-4381-9865-e697d3d85bca-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.684103 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.683979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.684103 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.683995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmxd\" (UniqueName: \"kubernetes.io/projected/2863783a-f9cd-4381-9865-e697d3d85bca-kube-api-access-ljmxd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.684319 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.684294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.684319 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.684314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.684420 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.684367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.686303 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.686283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2863783a-f9cd-4381-9865-e697d3d85bca-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.686508 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.686470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2863783a-f9cd-4381-9865-e697d3d85bca-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.694045 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.694020 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmxd\" (UniqueName: \"kubernetes.io/projected/2863783a-f9cd-4381-9865-e697d3d85bca-kube-api-access-ljmxd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sw57n\" (UID: \"2863783a-f9cd-4381-9865-e697d3d85bca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.726647 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.726614 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:41.872465 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:41.872434 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n"] Jul 16 17:46:41.875340 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:46:41.875311 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2863783a_f9cd_4381_9865_e697d3d85bca.slice/crio-cfa7c40436f3124037e535f68e5c84ff8cb3a77d884af7ba4af915a8d2704adb WatchSource:0}: Error finding container cfa7c40436f3124037e535f68e5c84ff8cb3a77d884af7ba4af915a8d2704adb: Status 404 returned error can't find the container with id cfa7c40436f3124037e535f68e5c84ff8cb3a77d884af7ba4af915a8d2704adb Jul 16 17:46:42.332913 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:42.332823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" event={"ID":"2863783a-f9cd-4381-9865-e697d3d85bca","Type":"ContainerStarted","Data":"6714b21062353808dfad558ba9a1aadc16362b2d7cd1d28e44a1039fcdfe9139"} Jul 16 17:46:42.332913 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:42.332887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" event={"ID":"2863783a-f9cd-4381-9865-e697d3d85bca","Type":"ContainerStarted","Data":"cfa7c40436f3124037e535f68e5c84ff8cb3a77d884af7ba4af915a8d2704adb"} Jul 16 17:46:42.357503 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:42.357449 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" podStartSLOduration=1.167868279 podStartE2EDuration="1.357433145s" totalImagesPullingTime="189.564866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:42.355348442 +0000 UTC m=+846.016912073" watchObservedRunningTime="2026-07-16 17:46:42.357433145 +0000 UTC m=+846.018996779" Jul 16 17:46:43.598655 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:43.598620 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:46:48.307130 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.307096 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd"] Jul 16 17:46:48.310666 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.310650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.313174 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.313145 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 17:46:48.321739 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.321692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd"] Jul 16 17:46:48.447471 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.447429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.447662 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.447502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxcn\" (UniqueName: \"kubernetes.io/projected/65a81e21-e810-469b-af2d-6a5c4636dbf9-kube-api-access-lqxcn\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.447662 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.447536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.447662 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.447571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.447662 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.447598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65a81e21-e810-469b-af2d-6a5c4636dbf9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.447858 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.447678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.548329 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.548291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.548525 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.548349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.548525 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.548391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxcn\" (UniqueName: \"kubernetes.io/projected/65a81e21-e810-469b-af2d-6a5c4636dbf9-kube-api-access-lqxcn\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.548525 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.548413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.548525 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.548438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.548525 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.548464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65a81e21-e810-469b-af2d-6a5c4636dbf9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.548826 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.548793 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.548874 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.548833 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.548978 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.548953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.550656 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.550630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65a81e21-e810-469b-af2d-6a5c4636dbf9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.550915 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.550898 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65a81e21-e810-469b-af2d-6a5c4636dbf9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.560132 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.560081 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxcn\" (UniqueName: \"kubernetes.io/projected/65a81e21-e810-469b-af2d-6a5c4636dbf9-kube-api-access-lqxcn\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd\" (UID: \"65a81e21-e810-469b-af2d-6a5c4636dbf9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.620352 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.620315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:48.772675 ip-10-0-128-217 kubenswrapper[2576]: W0716 17:46:48.772646 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65a81e21_e810_469b_af2d_6a5c4636dbf9.slice/crio-262fa14b30a77591d357308d7adac342371c261876b62eb68441cc3baf4315a8 WatchSource:0}: Error finding container 262fa14b30a77591d357308d7adac342371c261876b62eb68441cc3baf4315a8: Status 404 returned error can't find the container with id 262fa14b30a77591d357308d7adac342371c261876b62eb68441cc3baf4315a8 Jul 16 17:46:48.773397 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:48.773373 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd"] Jul 16 17:46:49.366492 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:49.366401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" event={"ID":"65a81e21-e810-469b-af2d-6a5c4636dbf9","Type":"ContainerStarted","Data":"3aae2ba30cac9e776895998dc8c4618963ec8be40264359bab0ee2c0ccfdb32f"} Jul 16 17:46:49.366492 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:49.366452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" event={"ID":"65a81e21-e810-469b-af2d-6a5c4636dbf9","Type":"ContainerStarted","Data":"262fa14b30a77591d357308d7adac342371c261876b62eb68441cc3baf4315a8"} Jul 16 17:46:49.393446 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:49.393398 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" podStartSLOduration=1.194542121 podStartE2EDuration="1.393380242s" totalImagesPullingTime="198.838121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:49.392181357 +0000 UTC m=+853.053744990" watchObservedRunningTime="2026-07-16 17:46:49.393380242 +0000 UTC m=+853.054943876" Jul 16 17:46:51.727480 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:51.727428 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:51.727971 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:51.727516 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:51.740236 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:51.740207 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:52.039507 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:52.039421 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:46:52.391996 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:52.391902 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sw57n" Jul 16 17:46:58.621098 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:58.621059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:58.621098 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:58.621105 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:58.633720 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:58.633668 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:46:59.423755 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:46:59.423720 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd" Jul 16 17:47:36.707012 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:47:36.706925 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:47:36.711481 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:47:36.711459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:48:34.798508 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:48:34.798469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:48:45.201082 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:48:45.201043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:48:55.016027 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:48:55.015994 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:50:40.896786 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:50:40.896688 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:51:21.838804 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:51:21.838775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:51:31.323642 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:51:31.323606 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:51:40.712587 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:51:40.712556 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:51:50.206495 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:51:50.206452 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:52:00.996609 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:52:00.996530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:52:10.804262 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:52:10.804228 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:52:21.306075 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:52:21.306039 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:52:31.501581 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:52:31.501548 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:52:36.737982 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:52:36.737948 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:52:36.739403 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:52:36.739384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:52:53.595303 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:52:53.595266 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:53:05.200477 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:53:05.200437 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:53:30.703320 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:53:30.703238 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:53:42.213362 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:53:42.213316 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:53:58.505037 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:53:58.504999 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:54:10.108995 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:54:10.108958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:54:21.603446 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:54:21.603413 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:56:26.617408 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:56:26.617370 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:56:38.419648 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:56:38.419565 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:56:47.314153 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:56:47.314119 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:56:59.929427 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:56:59.929388 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:57:09.713765 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:57:09.713728 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:57:17.314576 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:57:17.314533 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:57:27.700929 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:57:27.700886 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:57:36.768281 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:57:36.768251 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:57:36.769074 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:57:36.769045 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 17:57:38.012402 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:57:38.012363 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:57:45.833023 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:57:45.832990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:57:56.840668 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:57:56.840634 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:58:08.802638 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:58:08.802555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:58:16.621772 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:58:16.621740 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:58:34.697822 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:58:34.697786 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:58:44.501581 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:58:44.501538 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:58:54.098365 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:58:54.098329 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:59:04.930064 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:59:04.930028 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:59:15.007380 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:59:15.007346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:59:24.797782 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:59:24.797743 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:59:38.231455 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:59:38.231374 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:59:47.897287 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:59:47.897253 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 17:59:58.109469 ip-10-0-128-217 kubenswrapper[2576]: I0716 17:59:58.109431 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:00:07.927094 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:00:07.927050 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:00:18.094784 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:00:18.094745 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:00:28.896759 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:00:28.896724 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:00:40.293513 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:00:40.293469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:00:50.509262 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:00:50.509226 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:01:03.094569 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:01:03.094489 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:01:14.293372 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:01:14.293330 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:01:22.491328 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:01:22.491292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:01:35.294170 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:01:35.294135 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:01:49.695178 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:01:49.695141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:01:59.286250 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:01:59.286216 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:02:12.994831 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:02:12.994787 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:02:21.813572 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:02:21.813535 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:02:31.703840 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:02:31.703755 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:02:36.800714 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:02:36.800669 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:02:36.802134 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:02:36.802114 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:02:39.098199 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:02:39.098163 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:03:53.197216 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:03:53.197175 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:04:07.815922 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:04:07.815888 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:05:12.106422 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:05:12.106391 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:05:20.402451 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:05:20.402412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:06:35.908177 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:06:35.908143 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:06:51.112037 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:06:51.112005 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:07:36.831445 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:07:36.831413 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:07:36.833595 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:07:36.833572 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:08:06.014914 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:08:06.014876 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:09:44.988875 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:09:44.988841 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:09:57.492728 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:09:57.492683 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:10:08.493835 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:10:08.493745 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:10:27.493149 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:10:27.493109 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:11:07.702283 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:07.702242 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:11:12.603474 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.603432 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm"] Jul 16 18:11:12.607062 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.607039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.609423 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.609400 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7a2e4989\"/\"kube-root-ca.crt\"" Jul 16 18:11:12.609578 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.609559 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7a2e4989\"/\"openshift-service-ca.crt\"" Jul 16 18:11:12.609761 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.609742 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7a2e4989\"/\"default-dockercfg-ffmph\"" Jul 16 18:11:12.609972 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.609956 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7a2e4989\"/\"e2e-lifecycle-model-7a2e4989-kserve-self-signed-certs\"" Jul 16 18:11:12.615581 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.615559 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm"] Jul 16 18:11:12.696123 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.696069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-home\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.696321 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.696236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efffaa3d-6852-4091-a098-4fdd7a2359ad-tls-certs\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.696321 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.696291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-model-cache\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.696434 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.696341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-dshm\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.696434 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.696415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st5n5\" (UniqueName: \"kubernetes.io/projected/efffaa3d-6852-4091-a098-4fdd7a2359ad-kube-api-access-st5n5\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.696533 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.696478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-tmp-dir\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.797160 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.797121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efffaa3d-6852-4091-a098-4fdd7a2359ad-tls-certs\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.797160 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.797168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-model-cache\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.797400 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.797285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-dshm\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.797457 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.797398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st5n5\" (UniqueName: \"kubernetes.io/projected/efffaa3d-6852-4091-a098-4fdd7a2359ad-kube-api-access-st5n5\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.797513 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.797460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-tmp-dir\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.797572 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.797523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-home\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.797572 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.797530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-model-cache\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.797904 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.797875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-home\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.798017 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.797936 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-tmp-dir\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.799579 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.799553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-dshm\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.799813 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.799793 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efffaa3d-6852-4091-a098-4fdd7a2359ad-tls-certs\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.806927 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.806900 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st5n5\" (UniqueName: \"kubernetes.io/projected/efffaa3d-6852-4091-a098-4fdd7a2359ad-kube-api-access-st5n5\") pod \"e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:12.917055 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:12.917017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:13.064476 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:13.064450 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm"] Jul 16 18:11:13.066758 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:11:13.066727 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefffaa3d_6852_4091_a098_4fdd7a2359ad.slice/crio-372a4669f9a8222f2aa98e8921138da101dd4d1c8033d7586a249392a59c52aa WatchSource:0}: Error finding container 372a4669f9a8222f2aa98e8921138da101dd4d1c8033d7586a249392a59c52aa: Status 404 returned error can't find the container with id 372a4669f9a8222f2aa98e8921138da101dd4d1c8033d7586a249392a59c52aa Jul 16 18:11:13.140998 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:13.140960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" event={"ID":"efffaa3d-6852-4091-a098-4fdd7a2359ad","Type":"ContainerStarted","Data":"8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b"} Jul 16 18:11:13.141163 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:13.141007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" event={"ID":"efffaa3d-6852-4091-a098-4fdd7a2359ad","Type":"ContainerStarted","Data":"372a4669f9a8222f2aa98e8921138da101dd4d1c8033d7586a249392a59c52aa"} Jul 16 18:11:13.161652 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:13.161599 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" podStartSLOduration=1.161578816 podStartE2EDuration="1.161578816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:11:13.159822705 +0000 UTC m=+2316.821386338" watchObservedRunningTime="2026-07-16 18:11:13.161578816 +0000 UTC m=+2316.823142475" Jul 16 18:11:16.186599 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:16.186563 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:11:22.917552 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:22.917502 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:22.917552 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:22.917558 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:22.930306 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:22.930277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:23.207148 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:23.207071 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:11:28.505878 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:28.505795 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:11:49.002519 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:49.002485 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:11:58.090288 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:11:58.090252 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:12:03.159891 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:03.159856 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm"] Jul 16 18:12:18.180164 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.180125 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" podUID="efffaa3d-6852-4091-a098-4fdd7a2359ad" containerName="main" containerID="cri-o://8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b" gracePeriod=45 Jul 16 18:12:18.431890 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.431830 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:12:18.447455 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.447426 2576 generic.go:356] "Generic (PLEG): container finished" podID="efffaa3d-6852-4091-a098-4fdd7a2359ad" containerID="8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b" exitCode=0 Jul 16 18:12:18.447608 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.447468 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" Jul 16 18:12:18.447608 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.447510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" event={"ID":"efffaa3d-6852-4091-a098-4fdd7a2359ad","Type":"ContainerDied","Data":"8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b"} Jul 16 18:12:18.447608 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.447549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm" event={"ID":"efffaa3d-6852-4091-a098-4fdd7a2359ad","Type":"ContainerDied","Data":"372a4669f9a8222f2aa98e8921138da101dd4d1c8033d7586a249392a59c52aa"} Jul 16 18:12:18.447608 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.447566 2576 scope.go:117] "RemoveContainer" containerID="8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b" Jul 16 18:12:18.456608 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.456580 2576 scope.go:117] "RemoveContainer" containerID="8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b" Jul 16 18:12:18.456924 ip-10-0-128-217 kubenswrapper[2576]: E0716 18:12:18.456904 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b\": container with ID starting with 8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b not found: ID does not exist" containerID="8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b" Jul 16 18:12:18.457029 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.456932 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b"} err="failed to get container status \"8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b\": rpc error: code = NotFound desc = could not find container \"8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b\": container with ID starting with 8ba308bdd1bab8f150780f2854096f0bf28677d545317e68a2ef336e72e2385b not found: ID does not exist" Jul 16 18:12:18.505411 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.505382 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efffaa3d-6852-4091-a098-4fdd7a2359ad-tls-certs\") pod \"efffaa3d-6852-4091-a098-4fdd7a2359ad\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " Jul 16 18:12:18.505602 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.505421 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st5n5\" (UniqueName: \"kubernetes.io/projected/efffaa3d-6852-4091-a098-4fdd7a2359ad-kube-api-access-st5n5\") pod \"efffaa3d-6852-4091-a098-4fdd7a2359ad\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " Jul 16 18:12:18.505602 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.505459 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-model-cache\") pod \"efffaa3d-6852-4091-a098-4fdd7a2359ad\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " Jul 16 18:12:18.505602 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.505497 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-tmp-dir\") pod \"efffaa3d-6852-4091-a098-4fdd7a2359ad\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " Jul 16 18:12:18.505602 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.505534 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-home\") pod \"efffaa3d-6852-4091-a098-4fdd7a2359ad\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " Jul 16 18:12:18.505602 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.505558 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-dshm\") pod \"efffaa3d-6852-4091-a098-4fdd7a2359ad\" (UID: \"efffaa3d-6852-4091-a098-4fdd7a2359ad\") " Jul 16 18:12:18.505885 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.505811 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "efffaa3d-6852-4091-a098-4fdd7a2359ad" (UID: "efffaa3d-6852-4091-a098-4fdd7a2359ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:12:18.505885 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.505823 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efffaa3d-6852-4091-a098-4fdd7a2359ad" (UID: "efffaa3d-6852-4091-a098-4fdd7a2359ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:12:18.505885 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.505842 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-home" (OuterVolumeSpecName: "home") pod "efffaa3d-6852-4091-a098-4fdd7a2359ad" (UID: "efffaa3d-6852-4091-a098-4fdd7a2359ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:12:18.506078 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.506058 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-model-cache\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:12:18.506143 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.506083 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-tmp-dir\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:12:18.506143 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.506096 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-home\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:12:18.507839 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.507811 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efffaa3d-6852-4091-a098-4fdd7a2359ad-kube-api-access-st5n5" (OuterVolumeSpecName: "kube-api-access-st5n5") pod "efffaa3d-6852-4091-a098-4fdd7a2359ad" (UID: "efffaa3d-6852-4091-a098-4fdd7a2359ad"). InnerVolumeSpecName "kube-api-access-st5n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:12:18.508000 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.507983 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efffaa3d-6852-4091-a098-4fdd7a2359ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efffaa3d-6852-4091-a098-4fdd7a2359ad" (UID: "efffaa3d-6852-4091-a098-4fdd7a2359ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:12:18.508047 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.508024 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-dshm" (OuterVolumeSpecName: "dshm") pod "efffaa3d-6852-4091-a098-4fdd7a2359ad" (UID: "efffaa3d-6852-4091-a098-4fdd7a2359ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:12:18.607041 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.607002 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efffaa3d-6852-4091-a098-4fdd7a2359ad-dshm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:12:18.607041 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.607032 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efffaa3d-6852-4091-a098-4fdd7a2359ad-tls-certs\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:12:18.607041 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.607045 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st5n5\" (UniqueName: \"kubernetes.io/projected/efffaa3d-6852-4091-a098-4fdd7a2359ad-kube-api-access-st5n5\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:12:18.775442 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.775408 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm"] Jul 16 18:12:18.787577 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:18.787550 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7a2e4989/e2e-lifecycle-model-7a2e4989-kserve-55fbc48fcf-fxrfm"] Jul 16 18:12:20.771111 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:20.771082 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efffaa3d-6852-4091-a098-4fdd7a2359ad" path="/var/lib/kubelet/pods/efffaa3d-6852-4091-a098-4fdd7a2359ad/volumes" Jul 16 18:12:36.860569 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:36.860525 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:12:36.863906 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:36.863881 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:12:58.135290 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:12:58.135249 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:13:09.702992 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:13:09.702958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:14:07.602006 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:07.601968 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:14:16.709734 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:16.709683 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:14:40.294721 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.294621 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n"] Jul 16 18:14:40.295214 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.295029 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efffaa3d-6852-4091-a098-4fdd7a2359ad" containerName="main" Jul 16 18:14:40.295214 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.295043 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="efffaa3d-6852-4091-a098-4fdd7a2359ad" containerName="main" Jul 16 18:14:40.295214 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.295121 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="efffaa3d-6852-4091-a098-4fdd7a2359ad" containerName="main" Jul 16 18:14:40.298130 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.298115 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.301360 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.301336 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a1a1f5\"/\"openshift-service-ca.crt\"" Jul 16 18:14:40.301360 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.301356 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a1a1f5\"/\"default-dockercfg-72nf4\"" Jul 16 18:14:40.301777 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.301342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a1a1f5\"/\"test-model-a1a1f5-kserve-self-signed-certs\"" Jul 16 18:14:40.301777 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.301383 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a1a1f5\"/\"kube-root-ca.crt\"" Jul 16 18:14:40.307475 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.307214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-home\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.307475 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.307303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67494b55-1330-4637-835e-128350c9b678-tls-certs\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.307475 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.307390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-model-cache\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.307475 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.307420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-tmp-dir\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.307783 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.307479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-dshm\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.307783 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.307501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6pkt\" (UniqueName: \"kubernetes.io/projected/67494b55-1330-4637-835e-128350c9b678-kube-api-access-m6pkt\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.309304 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.309280 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n"] Jul 16 18:14:40.408863 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.408829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-model-cache\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.408863 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.408868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-tmp-dir\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.409082 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.408910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-dshm\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.409082 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.408930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6pkt\" (UniqueName: \"kubernetes.io/projected/67494b55-1330-4637-835e-128350c9b678-kube-api-access-m6pkt\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.409082 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.408960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-home\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.409082 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.409021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67494b55-1330-4637-835e-128350c9b678-tls-certs\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.409301 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.409276 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-model-cache\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.409301 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.409292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-tmp-dir\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.409471 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.409453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-home\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.411196 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.411176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-dshm\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.411595 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.411576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67494b55-1330-4637-835e-128350c9b678-tls-certs\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.420336 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.420311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6pkt\" (UniqueName: \"kubernetes.io/projected/67494b55-1330-4637-835e-128350c9b678-kube-api-access-m6pkt\") pod \"test-model-a1a1f5-kserve-55c8896d8c-9tk6n\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.607810 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.607717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:40.744736 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:40.744692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n"] Jul 16 18:14:40.747330 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:14:40.747304 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67494b55_1330_4637_835e_128350c9b678.slice/crio-8bfb45a22e538f00d929a8f399db8e05e4ead50309380e42197fd6f6c5686fa9 WatchSource:0}: Error finding container 8bfb45a22e538f00d929a8f399db8e05e4ead50309380e42197fd6f6c5686fa9: Status 404 returned error can't find the container with id 8bfb45a22e538f00d929a8f399db8e05e4ead50309380e42197fd6f6c5686fa9 Jul 16 18:14:41.115268 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:41.115229 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" event={"ID":"67494b55-1330-4637-835e-128350c9b678","Type":"ContainerStarted","Data":"2cf9558d4387e0fbcc6d1038be892016cb5c9c8661d87ef7a77e69ec83d39967"} Jul 16 18:14:41.115268 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:41.115276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" event={"ID":"67494b55-1330-4637-835e-128350c9b678","Type":"ContainerStarted","Data":"8bfb45a22e538f00d929a8f399db8e05e4ead50309380e42197fd6f6c5686fa9"} Jul 16 18:14:41.137376 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:41.137317 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" podStartSLOduration=1.13729992 podStartE2EDuration="1.13729992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:14:41.135965752 +0000 UTC m=+2524.797529401" watchObservedRunningTime="2026-07-16 18:14:41.13729992 +0000 UTC m=+2524.798863553" Jul 16 18:14:43.805967 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:43.805933 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:14:50.608512 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:50.608460 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:50.611064 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:50.608616 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:50.621110 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:50.621080 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:51.171839 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:51.171808 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:14:55.830837 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:14:55.830791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:15:26.406338 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.406304 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp"] Jul 16 18:15:26.409606 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.409576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.413424 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.413402 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-28bbcf\"/\"default-dockercfg-qrqn5\"" Jul 16 18:15:26.414304 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.414282 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-28bbcf\"/\"openshift-service-ca.crt\"" Jul 16 18:15:26.418508 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.418490 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-28bbcf\"/\"kube-root-ca.crt\"" Jul 16 18:15:26.420440 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.420422 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-28bbcf\"/\"test-model-28bbcf-kserve-self-signed-certs\"" Jul 16 18:15:26.423736 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.423692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp"] Jul 16 18:15:26.431460 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.431435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-model-cache\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.431576 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.431485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-tmp-dir\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.431576 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.431510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eda03c-5324-4842-9702-febd7b1732fa-tls-certs\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.431720 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.431633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpzw\" (UniqueName: \"kubernetes.io/projected/38eda03c-5324-4842-9702-febd7b1732fa-kube-api-access-5vpzw\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.431786 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.431723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-dshm\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.431786 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.431758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-home\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.532445 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.532403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-home\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.532656 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.532451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-model-cache\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.532656 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.532517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-tmp-dir\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.532656 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.532552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eda03c-5324-4842-9702-febd7b1732fa-tls-certs\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.532656 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.532610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpzw\" (UniqueName: \"kubernetes.io/projected/38eda03c-5324-4842-9702-febd7b1732fa-kube-api-access-5vpzw\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.532978 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.532667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-dshm\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.532978 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.532928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-home\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.532978 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.532952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-tmp-dir\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.533111 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.533016 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-model-cache\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.534989 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.534970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-dshm\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.535117 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.535101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eda03c-5324-4842-9702-febd7b1732fa-tls-certs\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.543531 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.543500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpzw\" (UniqueName: \"kubernetes.io/projected/38eda03c-5324-4842-9702-febd7b1732fa-kube-api-access-5vpzw\") pod \"test-model-28bbcf-kserve-6f4fc97f47-mkngp\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.719250 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.719154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:26.862096 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:26.862071 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp"] Jul 16 18:15:26.864304 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:15:26.864267 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38eda03c_5324_4842_9702_febd7b1732fa.slice/crio-52eaf2f779da801c1f33dac2bcdf8938ad56c1de0321d021c07004c0f2ead96c WatchSource:0}: Error finding container 52eaf2f779da801c1f33dac2bcdf8938ad56c1de0321d021c07004c0f2ead96c: Status 404 returned error can't find the container with id 52eaf2f779da801c1f33dac2bcdf8938ad56c1de0321d021c07004c0f2ead96c Jul 16 18:15:27.339741 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:27.339688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" event={"ID":"38eda03c-5324-4842-9702-febd7b1732fa","Type":"ContainerStarted","Data":"bcadefcc0f94eb6d536439ec390e86c9e6e54c1a6844fbbaf8d50049e741dc5e"} Jul 16 18:15:27.339741 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:27.339749 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" event={"ID":"38eda03c-5324-4842-9702-febd7b1732fa","Type":"ContainerStarted","Data":"52eaf2f779da801c1f33dac2bcdf8938ad56c1de0321d021c07004c0f2ead96c"} Jul 16 18:15:27.363718 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:27.363642 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" podStartSLOduration=1.363623187 podStartE2EDuration="1.363623187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:15:27.362464203 +0000 UTC m=+2571.024027837" watchObservedRunningTime="2026-07-16 18:15:27.363623187 +0000 UTC m=+2571.025186819" Jul 16 18:15:30.410500 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:30.410461 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:15:36.719327 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:36.719287 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:36.719871 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:36.719849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:36.732428 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:36.732398 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:37.401567 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:37.401538 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:15:44.698435 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:15:44.698401 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:16:12.381516 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:12.381478 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n"] Jul 16 18:16:13.374689 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:13.374655 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp"] Jul 16 18:16:17.892470 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:17.892414 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:16:27.402668 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.402621 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" podUID="67494b55-1330-4637-835e-128350c9b678" containerName="main" containerID="cri-o://2cf9558d4387e0fbcc6d1038be892016cb5c9c8661d87ef7a77e69ec83d39967" gracePeriod=45 Jul 16 18:16:27.634143 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.634114 2576 generic.go:356] "Generic (PLEG): container finished" podID="67494b55-1330-4637-835e-128350c9b678" containerID="2cf9558d4387e0fbcc6d1038be892016cb5c9c8661d87ef7a77e69ec83d39967" exitCode=0 Jul 16 18:16:27.634274 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.634184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" event={"ID":"67494b55-1330-4637-835e-128350c9b678","Type":"ContainerDied","Data":"2cf9558d4387e0fbcc6d1038be892016cb5c9c8661d87ef7a77e69ec83d39967"} Jul 16 18:16:27.651710 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.651670 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:16:27.706378 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706287 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-tmp-dir\") pod \"67494b55-1330-4637-835e-128350c9b678\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " Jul 16 18:16:27.706378 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706332 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6pkt\" (UniqueName: \"kubernetes.io/projected/67494b55-1330-4637-835e-128350c9b678-kube-api-access-m6pkt\") pod \"67494b55-1330-4637-835e-128350c9b678\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " Jul 16 18:16:27.706378 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706359 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67494b55-1330-4637-835e-128350c9b678-tls-certs\") pod \"67494b55-1330-4637-835e-128350c9b678\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " Jul 16 18:16:27.706654 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706441 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-home\") pod \"67494b55-1330-4637-835e-128350c9b678\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " Jul 16 18:16:27.706654 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706458 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-dshm\") pod \"67494b55-1330-4637-835e-128350c9b678\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " Jul 16 18:16:27.706654 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706478 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-model-cache\") pod \"67494b55-1330-4637-835e-128350c9b678\" (UID: \"67494b55-1330-4637-835e-128350c9b678\") " Jul 16 18:16:27.706844 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67494b55-1330-4637-835e-128350c9b678" (UID: "67494b55-1330-4637-835e-128350c9b678"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:16:27.706844 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706754 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-home" (OuterVolumeSpecName: "home") pod "67494b55-1330-4637-835e-128350c9b678" (UID: "67494b55-1330-4637-835e-128350c9b678"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:16:27.706844 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706835 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-tmp-dir\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:27.707001 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706851 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-home\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:27.707001 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.706828 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-model-cache" (OuterVolumeSpecName: "model-cache") pod "67494b55-1330-4637-835e-128350c9b678" (UID: "67494b55-1330-4637-835e-128350c9b678"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:16:27.708682 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.708655 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-dshm" (OuterVolumeSpecName: "dshm") pod "67494b55-1330-4637-835e-128350c9b678" (UID: "67494b55-1330-4637-835e-128350c9b678"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:16:27.708791 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.708718 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67494b55-1330-4637-835e-128350c9b678-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67494b55-1330-4637-835e-128350c9b678" (UID: "67494b55-1330-4637-835e-128350c9b678"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:16:27.708855 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.708835 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67494b55-1330-4637-835e-128350c9b678-kube-api-access-m6pkt" (OuterVolumeSpecName: "kube-api-access-m6pkt") pod "67494b55-1330-4637-835e-128350c9b678" (UID: "67494b55-1330-4637-835e-128350c9b678"). InnerVolumeSpecName "kube-api-access-m6pkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:16:27.807341 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.807298 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6pkt\" (UniqueName: \"kubernetes.io/projected/67494b55-1330-4637-835e-128350c9b678-kube-api-access-m6pkt\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:27.807341 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.807331 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67494b55-1330-4637-835e-128350c9b678-tls-certs\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:27.807341 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.807340 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-dshm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:27.807341 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:27.807349 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67494b55-1330-4637-835e-128350c9b678-model-cache\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:28.395228 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.395164 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" podUID="38eda03c-5324-4842-9702-febd7b1732fa" containerName="main" containerID="cri-o://bcadefcc0f94eb6d536439ec390e86c9e6e54c1a6844fbbaf8d50049e741dc5e" gracePeriod=45 Jul 16 18:16:28.641028 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.641003 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" Jul 16 18:16:28.641449 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.641002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n" event={"ID":"67494b55-1330-4637-835e-128350c9b678","Type":"ContainerDied","Data":"8bfb45a22e538f00d929a8f399db8e05e4ead50309380e42197fd6f6c5686fa9"} Jul 16 18:16:28.641449 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.641134 2576 scope.go:117] "RemoveContainer" containerID="2cf9558d4387e0fbcc6d1038be892016cb5c9c8661d87ef7a77e69ec83d39967" Jul 16 18:16:28.642491 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.642467 2576 generic.go:356] "Generic (PLEG): container finished" podID="38eda03c-5324-4842-9702-febd7b1732fa" containerID="bcadefcc0f94eb6d536439ec390e86c9e6e54c1a6844fbbaf8d50049e741dc5e" exitCode=0 Jul 16 18:16:28.642579 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.642520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" event={"ID":"38eda03c-5324-4842-9702-febd7b1732fa","Type":"ContainerDied","Data":"bcadefcc0f94eb6d536439ec390e86c9e6e54c1a6844fbbaf8d50049e741dc5e"} Jul 16 18:16:28.642579 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.642549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" event={"ID":"38eda03c-5324-4842-9702-febd7b1732fa","Type":"ContainerDied","Data":"52eaf2f779da801c1f33dac2bcdf8938ad56c1de0321d021c07004c0f2ead96c"} Jul 16 18:16:28.642579 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.642569 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52eaf2f779da801c1f33dac2bcdf8938ad56c1de0321d021c07004c0f2ead96c" Jul 16 18:16:28.648205 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.648147 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:16:28.688261 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.688229 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n"] Jul 16 18:16:28.706540 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.706511 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a1a1f5/test-model-a1a1f5-kserve-55c8896d8c-9tk6n"] Jul 16 18:16:28.715023 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715000 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-dshm\") pod \"38eda03c-5324-4842-9702-febd7b1732fa\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " Jul 16 18:16:28.715135 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715040 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-home\") pod \"38eda03c-5324-4842-9702-febd7b1732fa\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " Jul 16 18:16:28.715135 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715061 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eda03c-5324-4842-9702-febd7b1732fa-tls-certs\") pod \"38eda03c-5324-4842-9702-febd7b1732fa\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " Jul 16 18:16:28.715135 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715097 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-model-cache\") pod \"38eda03c-5324-4842-9702-febd7b1732fa\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " Jul 16 18:16:28.715291 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715146 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-tmp-dir\") pod \"38eda03c-5324-4842-9702-febd7b1732fa\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " Jul 16 18:16:28.715291 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715192 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vpzw\" (UniqueName: \"kubernetes.io/projected/38eda03c-5324-4842-9702-febd7b1732fa-kube-api-access-5vpzw\") pod \"38eda03c-5324-4842-9702-febd7b1732fa\" (UID: \"38eda03c-5324-4842-9702-febd7b1732fa\") " Jul 16 18:16:28.715394 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715362 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-home" (OuterVolumeSpecName: "home") pod "38eda03c-5324-4842-9702-febd7b1732fa" (UID: "38eda03c-5324-4842-9702-febd7b1732fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:16:28.715447 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715399 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "38eda03c-5324-4842-9702-febd7b1732fa" (UID: "38eda03c-5324-4842-9702-febd7b1732fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:16:28.715512 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715488 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38eda03c-5324-4842-9702-febd7b1732fa" (UID: "38eda03c-5324-4842-9702-febd7b1732fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:16:28.715512 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715501 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-home\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:28.715628 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.715514 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-model-cache\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:28.717366 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.717345 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38eda03c-5324-4842-9702-febd7b1732fa-kube-api-access-5vpzw" (OuterVolumeSpecName: "kube-api-access-5vpzw") pod "38eda03c-5324-4842-9702-febd7b1732fa" (UID: "38eda03c-5324-4842-9702-febd7b1732fa"). InnerVolumeSpecName "kube-api-access-5vpzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:16:28.717454 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.717400 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-dshm" (OuterVolumeSpecName: "dshm") pod "38eda03c-5324-4842-9702-febd7b1732fa" (UID: "38eda03c-5324-4842-9702-febd7b1732fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:16:28.717454 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.717428 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38eda03c-5324-4842-9702-febd7b1732fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38eda03c-5324-4842-9702-febd7b1732fa" (UID: "38eda03c-5324-4842-9702-febd7b1732fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:16:28.772059 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.772020 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67494b55-1330-4637-835e-128350c9b678" path="/var/lib/kubelet/pods/67494b55-1330-4637-835e-128350c9b678/volumes" Jul 16 18:16:28.816621 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.816583 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vpzw\" (UniqueName: \"kubernetes.io/projected/38eda03c-5324-4842-9702-febd7b1732fa-kube-api-access-5vpzw\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:28.816621 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.816615 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-dshm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:28.816621 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.816631 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eda03c-5324-4842-9702-febd7b1732fa-tls-certs\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:28.816894 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:28.816643 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eda03c-5324-4842-9702-febd7b1732fa-tmp-dir\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:16:29.648412 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:29.648387 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp" Jul 16 18:16:29.667662 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:29.667628 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp"] Jul 16 18:16:29.674621 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:29.674592 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-28bbcf/test-model-28bbcf-kserve-6f4fc97f47-mkngp"] Jul 16 18:16:30.772116 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:30.772082 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38eda03c-5324-4842-9702-febd7b1732fa" path="/var/lib/kubelet/pods/38eda03c-5324-4842-9702-febd7b1732fa/volumes" Jul 16 18:16:32.399301 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:32.399267 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:16:49.396151 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:16:49.396114 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:17:09.233013 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:17:09.232977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:17:22.297054 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:17:22.297017 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:17:36.894384 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:17:36.894309 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:17:36.897816 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:17:36.897795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:17:41.594266 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:17:41.594230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:18:12.691217 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:12.691184 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:18:31.616947 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:31.616911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:18:42.801688 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:42.801651 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:18:54.876016 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.875981 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll"] Jul 16 18:18:54.876409 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.876390 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67494b55-1330-4637-835e-128350c9b678" containerName="main" Jul 16 18:18:54.876409 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.876402 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="67494b55-1330-4637-835e-128350c9b678" containerName="main" Jul 16 18:18:54.876530 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.876414 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38eda03c-5324-4842-9702-febd7b1732fa" containerName="main" Jul 16 18:18:54.876530 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.876422 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="38eda03c-5324-4842-9702-febd7b1732fa" containerName="main" Jul 16 18:18:54.876530 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.876498 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="67494b55-1330-4637-835e-128350c9b678" containerName="main" Jul 16 18:18:54.876530 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.876511 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="38eda03c-5324-4842-9702-febd7b1732fa" containerName="main" Jul 16 18:18:54.879535 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.879509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:54.882112 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.882090 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a0389e\"/\"openshift-service-ca.crt\"" Jul 16 18:18:54.882901 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.882885 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a0389e\"/\"kube-root-ca.crt\"" Jul 16 18:18:54.883029 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.882955 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a0389e\"/\"default-dockercfg-26g5d\"" Jul 16 18:18:54.883084 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.883055 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a0389e\"/\"auth-test-model-a0389e-kserve-self-signed-certs\"" Jul 16 18:18:54.892610 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.892586 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll"] Jul 16 18:18:54.964927 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.964893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-dshm\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:54.964927 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.964930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b731367e-4841-40d9-825a-052e194e94f6-tls-certs\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:54.965157 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.964953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfq7z\" (UniqueName: \"kubernetes.io/projected/b731367e-4841-40d9-825a-052e194e94f6-kube-api-access-sfq7z\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:54.965157 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.965030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-tmp-dir\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:54.965157 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.965080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-model-cache\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:54.965266 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:54.965176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-home\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.065657 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.065613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-dshm\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.065657 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.065654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b731367e-4841-40d9-825a-052e194e94f6-tls-certs\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.065928 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.065678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfq7z\" (UniqueName: \"kubernetes.io/projected/b731367e-4841-40d9-825a-052e194e94f6-kube-api-access-sfq7z\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.065928 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.065738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-tmp-dir\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.065928 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.065890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-model-cache\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.066099 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.065980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-home\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.066154 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.066127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-tmp-dir\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.066268 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.066250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-home\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.066303 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.066247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-model-cache\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.067930 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.067910 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-dshm\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.068187 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.068167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b731367e-4841-40d9-825a-052e194e94f6-tls-certs\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.079031 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.079005 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfq7z\" (UniqueName: \"kubernetes.io/projected/b731367e-4841-40d9-825a-052e194e94f6-kube-api-access-sfq7z\") pod \"auth-test-model-a0389e-kserve-84c96847b7-kq5ll\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.188725 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.188672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:18:55.340274 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:55.340247 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll"] Jul 16 18:18:55.343022 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:18:55.342980 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb731367e_4841_40d9_825a_052e194e94f6.slice/crio-d2ebf96d8572d4ce39a681f5fb7124dd15b5b585d2e540a9289a001c7f7a5993 WatchSource:0}: Error finding container d2ebf96d8572d4ce39a681f5fb7124dd15b5b585d2e540a9289a001c7f7a5993: Status 404 returned error can't find the container with id d2ebf96d8572d4ce39a681f5fb7124dd15b5b585d2e540a9289a001c7f7a5993 Jul 16 18:18:56.335780 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:56.335739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" event={"ID":"b731367e-4841-40d9-825a-052e194e94f6","Type":"ContainerStarted","Data":"3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146"} Jul 16 18:18:56.335780 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:56.335788 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" event={"ID":"b731367e-4841-40d9-825a-052e194e94f6","Type":"ContainerStarted","Data":"d2ebf96d8572d4ce39a681f5fb7124dd15b5b585d2e540a9289a001c7f7a5993"} Jul 16 18:18:56.357654 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:56.357584 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" podStartSLOduration=2.357569245 podStartE2EDuration="2.357569245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:18:56.356141248 +0000 UTC m=+2780.017704881" watchObservedRunningTime="2026-07-16 18:18:56.357569245 +0000 UTC m=+2780.019132878" Jul 16 18:18:58.493755 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:18:58.493655 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:19:05.189330 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:05.189274 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:19:05.189330 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:05.189342 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:19:05.201982 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:05.201954 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:19:05.388134 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:05.388106 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:19:11.901944 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:11.901908 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt"] Jul 16 18:19:11.905556 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:11.905533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:11.908132 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:11.908114 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"kube-root-ca.crt\"" Jul 16 18:19:11.909228 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:11.909205 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"auth-test-model-bc54de-kserve-self-signed-certs\"" Jul 16 18:19:11.909327 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:11.909237 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"default-dockercfg-snlc4\"" Jul 16 18:19:11.909327 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:11.909316 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"openshift-service-ca.crt\"" Jul 16 18:19:11.915598 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:11.915567 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt"] Jul 16 18:19:12.006579 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.006543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-home\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.006579 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.006586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2rv\" (UniqueName: \"kubernetes.io/projected/499e1b78-1c73-4c5b-80bd-662b5130b8b7-kube-api-access-cj2rv\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.006860 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.006680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tmp-dir\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.006860 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.006796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-model-cache\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.006946 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.006863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-dshm\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.006946 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.006883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tls-certs\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.108013 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.107973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-home\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.108013 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.108011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2rv\" (UniqueName: \"kubernetes.io/projected/499e1b78-1c73-4c5b-80bd-662b5130b8b7-kube-api-access-cj2rv\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.108240 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.108064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tmp-dir\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.108240 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.108132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-model-cache\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.108240 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.108188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-dshm\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.108240 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.108213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tls-certs\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.108469 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.108449 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-home\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.108536 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.108519 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-model-cache\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.108574 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.108541 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tmp-dir\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.110562 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.110536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-dshm\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.110747 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.110732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tls-certs\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.120392 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.120369 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2rv\" (UniqueName: \"kubernetes.io/projected/499e1b78-1c73-4c5b-80bd-662b5130b8b7-kube-api-access-cj2rv\") pod \"auth-test-model-bc54de-kserve-6874d78699-jddzt\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.217284 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.217193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:12.354681 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.354652 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt"] Jul 16 18:19:12.356743 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:19:12.356714 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod499e1b78_1c73_4c5b_80bd_662b5130b8b7.slice/crio-ae8dc84e3c4c2dd8772575f2885b87c7d05445b0b0cbd24ce3046978d8437d20 WatchSource:0}: Error finding container ae8dc84e3c4c2dd8772575f2885b87c7d05445b0b0cbd24ce3046978d8437d20: Status 404 returned error can't find the container with id ae8dc84e3c4c2dd8772575f2885b87c7d05445b0b0cbd24ce3046978d8437d20 Jul 16 18:19:12.413303 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:12.413275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" event={"ID":"499e1b78-1c73-4c5b-80bd-662b5130b8b7","Type":"ContainerStarted","Data":"ae8dc84e3c4c2dd8772575f2885b87c7d05445b0b0cbd24ce3046978d8437d20"} Jul 16 18:19:13.419777 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:13.419744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" event={"ID":"499e1b78-1c73-4c5b-80bd-662b5130b8b7","Type":"ContainerStarted","Data":"0a3e91a1a4a45939b29cea36d50af06eae9657020cfaa06a2c6920dca44e2f60"} Jul 16 18:19:15.700347 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:15.700287 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" podStartSLOduration=4.700270058 podStartE2EDuration="4.700270058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:19:13.445153631 +0000 UTC m=+2797.106717269" watchObservedRunningTime="2026-07-16 18:19:15.700270058 +0000 UTC m=+2799.361833691" Jul 16 18:19:15.700872 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:15.700694 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:19:22.217821 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:22.217786 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:22.217821 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:22.217828 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:22.230435 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:22.230405 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:22.473591 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:22.473501 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:19:38.203258 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:38.203219 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:19:56.804362 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:19:56.804333 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:20:11.689416 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:20:11.689367 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:20:31.001845 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:20:31.001765 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:20:50.709270 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:20:50.709235 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:21:07.217097 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:07.217061 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:21:26.802337 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:26.802300 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:21:33.125945 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:33.125913 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll"] Jul 16 18:21:33.987134 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:33.987100 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt"] Jul 16 18:21:36.979573 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:36.979541 2576 scope.go:117] "RemoveContainer" containerID="bcadefcc0f94eb6d536439ec390e86c9e6e54c1a6844fbbaf8d50049e741dc5e" Jul 16 18:21:41.101738 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.101693 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg"] Jul 16 18:21:41.106729 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.106693 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.109071 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.109050 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a0389e\"/\"sub-test-model-a0389e-kserve-self-signed-certs\"" Jul 16 18:21:41.113994 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.113968 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg"] Jul 16 18:21:41.181631 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.181595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-tmp-dir\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.181857 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.181656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbb7\" (UniqueName: \"kubernetes.io/projected/ee2cef21-59b8-411f-98c1-2146552766fa-kube-api-access-8xbb7\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.181857 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.181686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2cef21-59b8-411f-98c1-2146552766fa-tls-certs\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.181857 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.181776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-model-cache\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.181857 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.181817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-dshm\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.182025 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.181901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-home\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.282634 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.282595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-tmp-dir\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.282861 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.282663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbb7\" (UniqueName: \"kubernetes.io/projected/ee2cef21-59b8-411f-98c1-2146552766fa-kube-api-access-8xbb7\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.282861 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.282726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2cef21-59b8-411f-98c1-2146552766fa-tls-certs\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.282861 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.282764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-model-cache\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.282861 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.282785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-dshm\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.282861 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.282855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-home\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.283148 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.283123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-tmp-dir\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.283196 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.283147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-model-cache\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.283236 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.283220 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-home\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.285029 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.285007 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-dshm\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.285348 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.285332 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2cef21-59b8-411f-98c1-2146552766fa-tls-certs\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.293101 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.293076 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbb7\" (UniqueName: \"kubernetes.io/projected/ee2cef21-59b8-411f-98c1-2146552766fa-kube-api-access-8xbb7\") pod \"sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.417139 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.417089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:41.574038 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:41.574000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg"] Jul 16 18:21:41.578509 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:21:41.578479 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee2cef21_59b8_411f_98c1_2146552766fa.slice/crio-181c0aa2553eded99c4358792780973b44924b76210e4a0ec28e61406434f942 WatchSource:0}: Error finding container 181c0aa2553eded99c4358792780973b44924b76210e4a0ec28e61406434f942: Status 404 returned error can't find the container with id 181c0aa2553eded99c4358792780973b44924b76210e4a0ec28e61406434f942 Jul 16 18:21:42.152272 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:42.152230 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" event={"ID":"ee2cef21-59b8-411f-98c1-2146552766fa","Type":"ContainerStarted","Data":"e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922"} Jul 16 18:21:42.152660 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:42.152282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" event={"ID":"ee2cef21-59b8-411f-98c1-2146552766fa","Type":"ContainerStarted","Data":"181c0aa2553eded99c4358792780973b44924b76210e4a0ec28e61406434f942"} Jul 16 18:21:42.172808 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:42.172752 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" podStartSLOduration=1.172737945 podStartE2EDuration="1.172737945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:21:42.170351558 +0000 UTC m=+2945.831915192" watchObservedRunningTime="2026-07-16 18:21:42.172737945 +0000 UTC m=+2945.834301607" Jul 16 18:21:45.597348 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:45.597309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:21:48.146151 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.146089 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" podUID="b731367e-4841-40d9-825a-052e194e94f6" containerName="main" containerID="cri-o://3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146" gracePeriod=45 Jul 16 18:21:48.400887 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.400821 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:21:48.551812 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.551770 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-tmp-dir\") pod \"b731367e-4841-40d9-825a-052e194e94f6\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " Jul 16 18:21:48.551986 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.551847 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-home\") pod \"b731367e-4841-40d9-825a-052e194e94f6\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " Jul 16 18:21:48.551986 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.551897 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-dshm\") pod \"b731367e-4841-40d9-825a-052e194e94f6\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " Jul 16 18:21:48.551986 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.551939 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-model-cache\") pod \"b731367e-4841-40d9-825a-052e194e94f6\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " Jul 16 18:21:48.551986 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.551977 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfq7z\" (UniqueName: \"kubernetes.io/projected/b731367e-4841-40d9-825a-052e194e94f6-kube-api-access-sfq7z\") pod \"b731367e-4841-40d9-825a-052e194e94f6\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " Jul 16 18:21:48.552242 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.552033 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b731367e-4841-40d9-825a-052e194e94f6-tls-certs\") pod \"b731367e-4841-40d9-825a-052e194e94f6\" (UID: \"b731367e-4841-40d9-825a-052e194e94f6\") " Jul 16 18:21:48.552242 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.552098 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b731367e-4841-40d9-825a-052e194e94f6" (UID: "b731367e-4841-40d9-825a-052e194e94f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:21:48.552242 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.552150 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-home" (OuterVolumeSpecName: "home") pod "b731367e-4841-40d9-825a-052e194e94f6" (UID: "b731367e-4841-40d9-825a-052e194e94f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:21:48.552242 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.552224 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "b731367e-4841-40d9-825a-052e194e94f6" (UID: "b731367e-4841-40d9-825a-052e194e94f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:21:48.552433 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.552345 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-model-cache\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:48.552433 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.552359 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-tmp-dir\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:48.552433 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.552369 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-home\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:48.554322 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.554292 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-dshm" (OuterVolumeSpecName: "dshm") pod "b731367e-4841-40d9-825a-052e194e94f6" (UID: "b731367e-4841-40d9-825a-052e194e94f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:21:48.554434 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.554330 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b731367e-4841-40d9-825a-052e194e94f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b731367e-4841-40d9-825a-052e194e94f6" (UID: "b731367e-4841-40d9-825a-052e194e94f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:21:48.554434 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.554412 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b731367e-4841-40d9-825a-052e194e94f6-kube-api-access-sfq7z" (OuterVolumeSpecName: "kube-api-access-sfq7z") pod "b731367e-4841-40d9-825a-052e194e94f6" (UID: "b731367e-4841-40d9-825a-052e194e94f6"). InnerVolumeSpecName "kube-api-access-sfq7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:21:48.653358 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.653262 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b731367e-4841-40d9-825a-052e194e94f6-dshm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:48.653358 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.653292 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfq7z\" (UniqueName: \"kubernetes.io/projected/b731367e-4841-40d9-825a-052e194e94f6-kube-api-access-sfq7z\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:48.653358 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:48.653303 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b731367e-4841-40d9-825a-052e194e94f6-tls-certs\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:49.007420 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.007330 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" podUID="499e1b78-1c73-4c5b-80bd-662b5130b8b7" containerName="main" containerID="cri-o://0a3e91a1a4a45939b29cea36d50af06eae9657020cfaa06a2c6920dca44e2f60" gracePeriod=45 Jul 16 18:21:49.186871 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.186841 2576 generic.go:356] "Generic (PLEG): container finished" podID="499e1b78-1c73-4c5b-80bd-662b5130b8b7" containerID="0a3e91a1a4a45939b29cea36d50af06eae9657020cfaa06a2c6920dca44e2f60" exitCode=0 Jul 16 18:21:49.187283 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.186921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" event={"ID":"499e1b78-1c73-4c5b-80bd-662b5130b8b7","Type":"ContainerDied","Data":"0a3e91a1a4a45939b29cea36d50af06eae9657020cfaa06a2c6920dca44e2f60"} Jul 16 18:21:49.188318 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.188293 2576 generic.go:356] "Generic (PLEG): container finished" podID="b731367e-4841-40d9-825a-052e194e94f6" containerID="3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146" exitCode=0 Jul 16 18:21:49.188458 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.188329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" event={"ID":"b731367e-4841-40d9-825a-052e194e94f6","Type":"ContainerDied","Data":"3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146"} Jul 16 18:21:49.188458 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.188358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" event={"ID":"b731367e-4841-40d9-825a-052e194e94f6","Type":"ContainerDied","Data":"d2ebf96d8572d4ce39a681f5fb7124dd15b5b585d2e540a9289a001c7f7a5993"} Jul 16 18:21:49.188458 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.188356 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll" Jul 16 18:21:49.188458 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.188375 2576 scope.go:117] "RemoveContainer" containerID="3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146" Jul 16 18:21:49.201042 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.200936 2576 scope.go:117] "RemoveContainer" containerID="3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146" Jul 16 18:21:49.201509 ip-10-0-128-217 kubenswrapper[2576]: E0716 18:21:49.201485 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146\": container with ID starting with 3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146 not found: ID does not exist" containerID="3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146" Jul 16 18:21:49.201636 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.201521 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146"} err="failed to get container status \"3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146\": rpc error: code = NotFound desc = could not find container \"3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146\": container with ID starting with 3301d4abd8b4c68f7c703f3c72cb8ce31c6333b7bae56bc0905eeb5e27c7a146 not found: ID does not exist" Jul 16 18:21:49.206979 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.206956 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll"] Jul 16 18:21:49.212072 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.212049 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/auth-test-model-a0389e-kserve-84c96847b7-kq5ll"] Jul 16 18:21:49.254203 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.254174 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:21:49.358907 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.358816 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-home\") pod \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " Jul 16 18:21:49.358907 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.358853 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj2rv\" (UniqueName: \"kubernetes.io/projected/499e1b78-1c73-4c5b-80bd-662b5130b8b7-kube-api-access-cj2rv\") pod \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " Jul 16 18:21:49.358907 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.358877 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-dshm\") pod \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " Jul 16 18:21:49.358907 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.358904 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-model-cache\") pod \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " Jul 16 18:21:49.359161 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.358996 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tmp-dir\") pod \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " Jul 16 18:21:49.359161 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.359068 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tls-certs\") pod \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\" (UID: \"499e1b78-1c73-4c5b-80bd-662b5130b8b7\") " Jul 16 18:21:49.359261 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.359168 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-home" (OuterVolumeSpecName: "home") pod "499e1b78-1c73-4c5b-80bd-662b5130b8b7" (UID: "499e1b78-1c73-4c5b-80bd-662b5130b8b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:21:49.359261 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.359177 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "499e1b78-1c73-4c5b-80bd-662b5130b8b7" (UID: "499e1b78-1c73-4c5b-80bd-662b5130b8b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:21:49.359369 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.359293 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "499e1b78-1c73-4c5b-80bd-662b5130b8b7" (UID: "499e1b78-1c73-4c5b-80bd-662b5130b8b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:21:49.359428 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.359405 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-home\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:49.359428 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.359416 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-model-cache\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:49.359428 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.359426 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tmp-dir\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:49.361238 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.361219 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "499e1b78-1c73-4c5b-80bd-662b5130b8b7" (UID: "499e1b78-1c73-4c5b-80bd-662b5130b8b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:21:49.361507 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.361481 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499e1b78-1c73-4c5b-80bd-662b5130b8b7-kube-api-access-cj2rv" (OuterVolumeSpecName: "kube-api-access-cj2rv") pod "499e1b78-1c73-4c5b-80bd-662b5130b8b7" (UID: "499e1b78-1c73-4c5b-80bd-662b5130b8b7"). InnerVolumeSpecName "kube-api-access-cj2rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:21:49.361507 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.361483 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-dshm" (OuterVolumeSpecName: "dshm") pod "499e1b78-1c73-4c5b-80bd-662b5130b8b7" (UID: "499e1b78-1c73-4c5b-80bd-662b5130b8b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:21:49.460953 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.460854 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499e1b78-1c73-4c5b-80bd-662b5130b8b7-tls-certs\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:49.460953 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.460888 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj2rv\" (UniqueName: \"kubernetes.io/projected/499e1b78-1c73-4c5b-80bd-662b5130b8b7-kube-api-access-cj2rv\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:49.460953 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:49.460904 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/499e1b78-1c73-4c5b-80bd-662b5130b8b7-dshm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:21:50.194960 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:50.194933 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" Jul 16 18:21:50.195412 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:50.194986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt" event={"ID":"499e1b78-1c73-4c5b-80bd-662b5130b8b7","Type":"ContainerDied","Data":"ae8dc84e3c4c2dd8772575f2885b87c7d05445b0b0cbd24ce3046978d8437d20"} Jul 16 18:21:50.195412 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:50.195034 2576 scope.go:117] "RemoveContainer" containerID="0a3e91a1a4a45939b29cea36d50af06eae9657020cfaa06a2c6920dca44e2f60" Jul 16 18:21:50.218269 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:50.218237 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt"] Jul 16 18:21:50.223492 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:50.223471 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/auth-test-model-bc54de-kserve-6874d78699-jddzt"] Jul 16 18:21:50.774335 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:50.774291 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="499e1b78-1c73-4c5b-80bd-662b5130b8b7" path="/var/lib/kubelet/pods/499e1b78-1c73-4c5b-80bd-662b5130b8b7/volumes" Jul 16 18:21:50.774977 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:50.774952 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b731367e-4841-40d9-825a-052e194e94f6" path="/var/lib/kubelet/pods/b731367e-4841-40d9-825a-052e194e94f6/volumes" Jul 16 18:21:51.417461 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:51.417419 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:51.417461 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:51.417467 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:51.430038 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:51.430014 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:21:52.218097 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:21:52.218070 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:22:05.597784 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.597690 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc"] Jul 16 18:22:05.598244 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.598225 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b731367e-4841-40d9-825a-052e194e94f6" containerName="main" Jul 16 18:22:05.598320 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.598249 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b731367e-4841-40d9-825a-052e194e94f6" containerName="main" Jul 16 18:22:05.598320 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.598273 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="499e1b78-1c73-4c5b-80bd-662b5130b8b7" containerName="main" Jul 16 18:22:05.598320 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.598282 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="499e1b78-1c73-4c5b-80bd-662b5130b8b7" containerName="main" Jul 16 18:22:05.598471 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.598404 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b731367e-4841-40d9-825a-052e194e94f6" containerName="main" Jul 16 18:22:05.598471 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.598419 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="499e1b78-1c73-4c5b-80bd-662b5130b8b7" containerName="main" Jul 16 18:22:05.602629 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.602608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.605361 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.605342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"default-dockercfg-snlc4\"" Jul 16 18:22:05.605478 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.605373 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"kube-root-ca.crt\"" Jul 16 18:22:05.606094 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.606077 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"openshift-service-ca.crt\"" Jul 16 18:22:05.606225 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.606210 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"sub-test-model-bc54de-kserve-self-signed-certs\"" Jul 16 18:22:05.610930 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.610911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc"] Jul 16 18:22:05.713476 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.713441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-dshm\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.713476 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.713480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29028a91-630d-498d-8cf6-5f55a52a632d-tls-certs\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.713753 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.713610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-home\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.713753 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.713639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-tmp-dir\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.713753 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.713668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-model-cache\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.713753 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.713689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgrrm\" (UniqueName: \"kubernetes.io/projected/29028a91-630d-498d-8cf6-5f55a52a632d-kube-api-access-dgrrm\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.815136 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.815103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-home\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.815136 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.815132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-tmp-dir\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.815373 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.815157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-model-cache\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.815373 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.815274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrrm\" (UniqueName: \"kubernetes.io/projected/29028a91-630d-498d-8cf6-5f55a52a632d-kube-api-access-dgrrm\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.815486 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.815386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-dshm\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.815486 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.815416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29028a91-630d-498d-8cf6-5f55a52a632d-tls-certs\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.815486 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.815461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-model-cache\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.815630 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.815515 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-tmp-dir\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.815630 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.815551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-home\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.817510 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.817491 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-dshm\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.817898 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.817878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29028a91-630d-498d-8cf6-5f55a52a632d-tls-certs\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.825775 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.825751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrrm\" (UniqueName: \"kubernetes.io/projected/29028a91-630d-498d-8cf6-5f55a52a632d-kube-api-access-dgrrm\") pod \"sub-test-model-bc54de-kserve-5b478dbf85-2zsrc\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:05.912741 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:05.912672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:06.043481 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:06.043459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc"] Jul 16 18:22:06.045187 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:22:06.045159 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29028a91_630d_498d_8cf6_5f55a52a632d.slice/crio-864db796d79bb6f54aae30d351f68d3e07009197d49d9c11c5fb861d25b1fb88 WatchSource:0}: Error finding container 864db796d79bb6f54aae30d351f68d3e07009197d49d9c11c5fb861d25b1fb88: Status 404 returned error can't find the container with id 864db796d79bb6f54aae30d351f68d3e07009197d49d9c11c5fb861d25b1fb88 Jul 16 18:22:06.272664 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:06.272573 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" event={"ID":"29028a91-630d-498d-8cf6-5f55a52a632d","Type":"ContainerStarted","Data":"53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597"} Jul 16 18:22:06.272664 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:06.272619 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" event={"ID":"29028a91-630d-498d-8cf6-5f55a52a632d","Type":"ContainerStarted","Data":"864db796d79bb6f54aae30d351f68d3e07009197d49d9c11c5fb861d25b1fb88"} Jul 16 18:22:06.292542 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:06.292484 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" podStartSLOduration=1.292468027 podStartE2EDuration="1.292468027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:22:06.290775387 +0000 UTC m=+2969.952339023" watchObservedRunningTime="2026-07-16 18:22:06.292468027 +0000 UTC m=+2969.954031660" Jul 16 18:22:09.500965 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:09.500930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:22:15.913442 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:15.913403 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:15.913989 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:15.913473 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:15.926038 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:15.926009 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:16.339981 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:16.339890 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:22:25.605225 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:25.605176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:22:36.924985 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:36.924955 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:22:36.928680 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:36.928660 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8ksg_3ce7cd9b-b3ce-4fd5-83ba-9fbff3f7a607/ovn-acl-logging/0.log" Jul 16 18:22:43.894488 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:43.894448 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:22:50.688280 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:50.688249 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg"] Jul 16 18:22:51.600957 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:22:51.600918 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc"] Jul 16 18:23:02.589081 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:02.589046 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:23:04.903037 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.903002 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5"] Jul 16 18:23:04.906904 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.906882 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:04.909825 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.909802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a0389e\"/\"rate-test-model-a0389e-kserve-self-signed-certs\"" Jul 16 18:23:04.918207 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.918182 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5"] Jul 16 18:23:04.951854 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.951819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-dshm\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:04.952018 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.951873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-home\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:04.952018 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.951953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97d62\" (UniqueName: \"kubernetes.io/projected/bd572114-bd0e-4dbc-a04b-5cd05336e333-kube-api-access-97d62\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:04.952118 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.952017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-tmp-dir\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:04.952118 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.952083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd572114-bd0e-4dbc-a04b-5cd05336e333-tls-certs\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:04.952216 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:04.952132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-model-cache\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.053579 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.053538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-tmp-dir\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.053579 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.053578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd572114-bd0e-4dbc-a04b-5cd05336e333-tls-certs\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.053881 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.053596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-model-cache\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.053881 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.053641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-dshm\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.053881 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.053678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-home\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.053881 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.053761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97d62\" (UniqueName: \"kubernetes.io/projected/bd572114-bd0e-4dbc-a04b-5cd05336e333-kube-api-access-97d62\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.054107 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.054041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-tmp-dir\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.054107 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.054051 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-model-cache\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.054180 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.054145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-home\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.056004 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.055975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-dshm\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.056238 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.056224 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd572114-bd0e-4dbc-a04b-5cd05336e333-tls-certs\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.063585 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.063562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97d62\" (UniqueName: \"kubernetes.io/projected/bd572114-bd0e-4dbc-a04b-5cd05336e333-kube-api-access-97d62\") pod \"rate-test-model-a0389e-kserve-678595c595-xk5s5\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.216159 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.216067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:05.351102 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.351071 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5"] Jul 16 18:23:05.353195 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:23:05.353166 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd572114_bd0e_4dbc_a04b_5cd05336e333.slice/crio-ac1ef92b500f3fe6db00f01dc3dbff1929a5faf7ec918e474f7a009c049a2a8f WatchSource:0}: Error finding container ac1ef92b500f3fe6db00f01dc3dbff1929a5faf7ec918e474f7a009c049a2a8f: Status 404 returned error can't find the container with id ac1ef92b500f3fe6db00f01dc3dbff1929a5faf7ec918e474f7a009c049a2a8f Jul 16 18:23:05.562880 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.562777 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" event={"ID":"bd572114-bd0e-4dbc-a04b-5cd05336e333","Type":"ContainerStarted","Data":"54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6"} Jul 16 18:23:05.562880 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.562836 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" event={"ID":"bd572114-bd0e-4dbc-a04b-5cd05336e333","Type":"ContainerStarted","Data":"ac1ef92b500f3fe6db00f01dc3dbff1929a5faf7ec918e474f7a009c049a2a8f"} Jul 16 18:23:05.582065 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.582016 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" podStartSLOduration=1.5819981730000001 podStartE2EDuration="1.581998173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:23:05.580603969 +0000 UTC m=+3029.242167603" watchObservedRunningTime="2026-07-16 18:23:05.581998173 +0000 UTC m=+3029.243561850" Jul 16 18:23:05.709083 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.709039 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" podUID="ee2cef21-59b8-411f-98c1-2146552766fa" containerName="main" containerID="cri-o://e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922" gracePeriod=45 Jul 16 18:23:05.952570 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:05.952547 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:23:06.061506 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.061470 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-tmp-dir\") pod \"ee2cef21-59b8-411f-98c1-2146552766fa\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " Jul 16 18:23:06.061506 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.061504 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xbb7\" (UniqueName: \"kubernetes.io/projected/ee2cef21-59b8-411f-98c1-2146552766fa-kube-api-access-8xbb7\") pod \"ee2cef21-59b8-411f-98c1-2146552766fa\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " Jul 16 18:23:06.061737 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.061571 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-model-cache\") pod \"ee2cef21-59b8-411f-98c1-2146552766fa\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " Jul 16 18:23:06.061737 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.061598 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-home\") pod \"ee2cef21-59b8-411f-98c1-2146552766fa\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " Jul 16 18:23:06.061846 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.061756 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-dshm\") pod \"ee2cef21-59b8-411f-98c1-2146552766fa\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " Jul 16 18:23:06.061846 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.061801 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee2cef21-59b8-411f-98c1-2146552766fa" (UID: "ee2cef21-59b8-411f-98c1-2146552766fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:23:06.061846 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.061833 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2cef21-59b8-411f-98c1-2146552766fa-tls-certs\") pod \"ee2cef21-59b8-411f-98c1-2146552766fa\" (UID: \"ee2cef21-59b8-411f-98c1-2146552766fa\") " Jul 16 18:23:06.062047 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.061838 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee2cef21-59b8-411f-98c1-2146552766fa" (UID: "ee2cef21-59b8-411f-98c1-2146552766fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:23:06.062047 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.061905 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-home" (OuterVolumeSpecName: "home") pod "ee2cef21-59b8-411f-98c1-2146552766fa" (UID: "ee2cef21-59b8-411f-98c1-2146552766fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:23:06.062167 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.062149 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-tmp-dir\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:06.062207 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.062172 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-model-cache\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:06.062207 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.062186 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-home\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:06.063952 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.063925 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2cef21-59b8-411f-98c1-2146552766fa-kube-api-access-8xbb7" (OuterVolumeSpecName: "kube-api-access-8xbb7") pod "ee2cef21-59b8-411f-98c1-2146552766fa" (UID: "ee2cef21-59b8-411f-98c1-2146552766fa"). InnerVolumeSpecName "kube-api-access-8xbb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:23:06.064206 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.064178 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-dshm" (OuterVolumeSpecName: "dshm") pod "ee2cef21-59b8-411f-98c1-2146552766fa" (UID: "ee2cef21-59b8-411f-98c1-2146552766fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:23:06.064206 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.064197 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee2cef21-59b8-411f-98c1-2146552766fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee2cef21-59b8-411f-98c1-2146552766fa" (UID: "ee2cef21-59b8-411f-98c1-2146552766fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:23:06.163357 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.163324 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2cef21-59b8-411f-98c1-2146552766fa-tls-certs\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:06.163357 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.163352 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbb7\" (UniqueName: \"kubernetes.io/projected/ee2cef21-59b8-411f-98c1-2146552766fa-kube-api-access-8xbb7\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:06.163357 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.163361 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2cef21-59b8-411f-98c1-2146552766fa-dshm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:06.569626 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.569596 2576 generic.go:356] "Generic (PLEG): container finished" podID="ee2cef21-59b8-411f-98c1-2146552766fa" containerID="e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922" exitCode=0 Jul 16 18:23:06.569814 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.569642 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" Jul 16 18:23:06.569814 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.569693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" event={"ID":"ee2cef21-59b8-411f-98c1-2146552766fa","Type":"ContainerDied","Data":"e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922"} Jul 16 18:23:06.569814 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.569753 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg" event={"ID":"ee2cef21-59b8-411f-98c1-2146552766fa","Type":"ContainerDied","Data":"181c0aa2553eded99c4358792780973b44924b76210e4a0ec28e61406434f942"} Jul 16 18:23:06.569814 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.569771 2576 scope.go:117] "RemoveContainer" containerID="e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922" Jul 16 18:23:06.578339 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.578323 2576 scope.go:117] "RemoveContainer" containerID="e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922" Jul 16 18:23:06.578555 ip-10-0-128-217 kubenswrapper[2576]: E0716 18:23:06.578537 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922\": container with ID starting with e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922 not found: ID does not exist" containerID="e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922" Jul 16 18:23:06.578598 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.578564 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922"} err="failed to get container status \"e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922\": rpc error: code = NotFound desc = could not find container \"e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922\": container with ID starting with e65cdc58374f57f5885014041872097676dec62f2605c2f0a23d228de2efa922 not found: ID does not exist" Jul 16 18:23:06.592141 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.592116 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg"] Jul 16 18:23:06.596711 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.596684 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/sub-test-model-a0389e-kserve-5cc6446bdc-mnsgg"] Jul 16 18:23:06.621930 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.621877 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" podUID="29028a91-630d-498d-8cf6-5f55a52a632d" containerName="main" containerID="cri-o://53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597" gracePeriod=45 Jul 16 18:23:06.778296 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.775249 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee2cef21-59b8-411f-98c1-2146552766fa" path="/var/lib/kubelet/pods/ee2cef21-59b8-411f-98c1-2146552766fa/volumes" Jul 16 18:23:06.888307 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.888278 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:23:06.968400 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968368 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-home\") pod \"29028a91-630d-498d-8cf6-5f55a52a632d\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " Jul 16 18:23:06.968906 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968415 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgrrm\" (UniqueName: \"kubernetes.io/projected/29028a91-630d-498d-8cf6-5f55a52a632d-kube-api-access-dgrrm\") pod \"29028a91-630d-498d-8cf6-5f55a52a632d\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " Jul 16 18:23:06.968906 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968443 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29028a91-630d-498d-8cf6-5f55a52a632d-tls-certs\") pod \"29028a91-630d-498d-8cf6-5f55a52a632d\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " Jul 16 18:23:06.968906 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968463 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-model-cache\") pod \"29028a91-630d-498d-8cf6-5f55a52a632d\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " Jul 16 18:23:06.968906 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968484 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-dshm\") pod \"29028a91-630d-498d-8cf6-5f55a52a632d\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " Jul 16 18:23:06.968906 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968575 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-tmp-dir\") pod \"29028a91-630d-498d-8cf6-5f55a52a632d\" (UID: \"29028a91-630d-498d-8cf6-5f55a52a632d\") " Jul 16 18:23:06.968906 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968755 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-home" (OuterVolumeSpecName: "home") pod "29028a91-630d-498d-8cf6-5f55a52a632d" (UID: "29028a91-630d-498d-8cf6-5f55a52a632d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:23:06.968906 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968793 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-model-cache" (OuterVolumeSpecName: "model-cache") pod "29028a91-630d-498d-8cf6-5f55a52a632d" (UID: "29028a91-630d-498d-8cf6-5f55a52a632d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:23:06.969160 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968947 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-home\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:06.969160 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968967 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-model-cache\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:06.969160 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.968990 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29028a91-630d-498d-8cf6-5f55a52a632d" (UID: "29028a91-630d-498d-8cf6-5f55a52a632d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:23:06.970724 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.970671 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-dshm" (OuterVolumeSpecName: "dshm") pod "29028a91-630d-498d-8cf6-5f55a52a632d" (UID: "29028a91-630d-498d-8cf6-5f55a52a632d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:23:06.970834 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.970765 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29028a91-630d-498d-8cf6-5f55a52a632d-kube-api-access-dgrrm" (OuterVolumeSpecName: "kube-api-access-dgrrm") pod "29028a91-630d-498d-8cf6-5f55a52a632d" (UID: "29028a91-630d-498d-8cf6-5f55a52a632d"). InnerVolumeSpecName "kube-api-access-dgrrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:23:06.970834 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:06.970774 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29028a91-630d-498d-8cf6-5f55a52a632d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29028a91-630d-498d-8cf6-5f55a52a632d" (UID: "29028a91-630d-498d-8cf6-5f55a52a632d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:23:07.070134 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.070101 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-tmp-dir\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:07.070134 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.070125 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgrrm\" (UniqueName: \"kubernetes.io/projected/29028a91-630d-498d-8cf6-5f55a52a632d-kube-api-access-dgrrm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:07.070309 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.070139 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29028a91-630d-498d-8cf6-5f55a52a632d-tls-certs\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:07.070309 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.070154 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29028a91-630d-498d-8cf6-5f55a52a632d-dshm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:23:07.576129 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.576096 2576 generic.go:356] "Generic (PLEG): container finished" podID="29028a91-630d-498d-8cf6-5f55a52a632d" containerID="53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597" exitCode=0 Jul 16 18:23:07.576319 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.576139 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" Jul 16 18:23:07.576319 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.576183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" event={"ID":"29028a91-630d-498d-8cf6-5f55a52a632d","Type":"ContainerDied","Data":"53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597"} Jul 16 18:23:07.576319 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.576228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc" event={"ID":"29028a91-630d-498d-8cf6-5f55a52a632d","Type":"ContainerDied","Data":"864db796d79bb6f54aae30d351f68d3e07009197d49d9c11c5fb861d25b1fb88"} Jul 16 18:23:07.576319 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.576247 2576 scope.go:117] "RemoveContainer" containerID="53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597" Jul 16 18:23:07.584909 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.584890 2576 scope.go:117] "RemoveContainer" containerID="53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597" Jul 16 18:23:07.585150 ip-10-0-128-217 kubenswrapper[2576]: E0716 18:23:07.585132 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597\": container with ID starting with 53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597 not found: ID does not exist" containerID="53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597" Jul 16 18:23:07.585213 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.585160 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597"} err="failed to get container status \"53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597\": rpc error: code = NotFound desc = could not find container \"53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597\": container with ID starting with 53eb87c63d4a68b0d47c170b50384103b3f412e8d2ee78bb94139b4837d9d597 not found: ID does not exist" Jul 16 18:23:07.604366 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.604344 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc"] Jul 16 18:23:07.613615 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:07.613590 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/sub-test-model-bc54de-kserve-5b478dbf85-2zsrc"] Jul 16 18:23:08.771757 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:08.771722 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29028a91-630d-498d-8cf6-5f55a52a632d" path="/var/lib/kubelet/pods/29028a91-630d-498d-8cf6-5f55a52a632d/volumes" Jul 16 18:23:15.216308 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:15.216268 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:15.216828 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:15.216327 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:15.229053 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:15.229026 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:15.629611 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:15.629528 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:23:16.390884 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:16.390851 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:23:43.893811 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.893756 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd"] Jul 16 18:23:43.894337 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.894111 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2cef21-59b8-411f-98c1-2146552766fa" containerName="main" Jul 16 18:23:43.894337 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.894122 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2cef21-59b8-411f-98c1-2146552766fa" containerName="main" Jul 16 18:23:43.894337 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.894135 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29028a91-630d-498d-8cf6-5f55a52a632d" containerName="main" Jul 16 18:23:43.894337 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.894140 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="29028a91-630d-498d-8cf6-5f55a52a632d" containerName="main" Jul 16 18:23:43.894337 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.894216 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee2cef21-59b8-411f-98c1-2146552766fa" containerName="main" Jul 16 18:23:43.894337 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.894226 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="29028a91-630d-498d-8cf6-5f55a52a632d" containerName="main" Jul 16 18:23:43.898577 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.898557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:43.901036 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.901013 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"openshift-service-ca.crt\"" Jul 16 18:23:43.901892 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.901873 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"rate-test-model-bc54de-kserve-self-signed-certs\"" Jul 16 18:23:43.902001 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.901927 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"default-dockercfg-snlc4\"" Jul 16 18:23:43.902068 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.902011 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc54de\"/\"kube-root-ca.crt\"" Jul 16 18:23:43.906287 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:43.906265 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd"] Jul 16 18:23:44.007734 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.007664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-model-cache\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.007734 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.007721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tls-certs\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.007982 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.007764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-home\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.007982 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.007849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tmp-dir\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.007982 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.007887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b82r\" (UniqueName: \"kubernetes.io/projected/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-kube-api-access-8b82r\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.007982 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.007946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-dshm\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.108995 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.108955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-dshm\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.109204 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.109031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-model-cache\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.109204 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.109051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tls-certs\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.109204 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.109075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-home\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.109204 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.109112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tmp-dir\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.109204 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.109130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b82r\" (UniqueName: \"kubernetes.io/projected/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-kube-api-access-8b82r\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.109525 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.109496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-home\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.109583 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.109524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tmp-dir\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.109583 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.109563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-model-cache\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.111332 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.111298 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-dshm\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.111502 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.111484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tls-certs\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.118439 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.118403 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b82r\" (UniqueName: \"kubernetes.io/projected/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-kube-api-access-8b82r\") pod \"rate-test-model-bc54de-kserve-55f687cf5-rhbbd\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.209178 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.209081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:44.343782 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.343666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd"] Jul 16 18:23:44.346577 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:23:44.346549 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9ada32a_e4f0_4f85_9f82_00bc1e80b034.slice/crio-2c5d3616baa859b0626c3ec2bfe9cb7d7efe3cf4ab593fc76bf29761ee281d43 WatchSource:0}: Error finding container 2c5d3616baa859b0626c3ec2bfe9cb7d7efe3cf4ab593fc76bf29761ee281d43: Status 404 returned error can't find the container with id 2c5d3616baa859b0626c3ec2bfe9cb7d7efe3cf4ab593fc76bf29761ee281d43 Jul 16 18:23:44.756972 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.756935 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" event={"ID":"a9ada32a-e4f0-4f85-9f82-00bc1e80b034","Type":"ContainerStarted","Data":"f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197"} Jul 16 18:23:44.757159 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.756982 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" event={"ID":"a9ada32a-e4f0-4f85-9f82-00bc1e80b034","Type":"ContainerStarted","Data":"2c5d3616baa859b0626c3ec2bfe9cb7d7efe3cf4ab593fc76bf29761ee281d43"} Jul 16 18:23:44.776294 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:44.776241 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" podStartSLOduration=1.776224339 podStartE2EDuration="1.776224339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:23:44.774384904 +0000 UTC m=+3068.435948538" watchObservedRunningTime="2026-07-16 18:23:44.776224339 +0000 UTC m=+3068.437787972" Jul 16 18:23:47.998002 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:47.997963 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:23:54.209781 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:54.209742 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:54.210206 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:54.210084 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:54.222219 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:54.222194 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:23:54.817679 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:23:54.817651 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:24:03.791159 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:03.791115 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:24:17.797159 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:17.797117 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:24:24.202930 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:24.202893 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5"] Jul 16 18:24:25.077171 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:25.077139 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd"] Jul 16 18:24:35.702121 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:35.702083 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:24:39.223693 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.223651 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" podUID="bd572114-bd0e-4dbc-a04b-5cd05336e333" containerName="main" containerID="cri-o://54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6" gracePeriod=45 Jul 16 18:24:39.470998 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.470972 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:24:39.531240 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531157 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-home\") pod \"bd572114-bd0e-4dbc-a04b-5cd05336e333\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " Jul 16 18:24:39.531240 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531202 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-dshm\") pod \"bd572114-bd0e-4dbc-a04b-5cd05336e333\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " Jul 16 18:24:39.531461 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531248 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-model-cache\") pod \"bd572114-bd0e-4dbc-a04b-5cd05336e333\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " Jul 16 18:24:39.531461 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531295 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97d62\" (UniqueName: \"kubernetes.io/projected/bd572114-bd0e-4dbc-a04b-5cd05336e333-kube-api-access-97d62\") pod \"bd572114-bd0e-4dbc-a04b-5cd05336e333\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " Jul 16 18:24:39.531461 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531369 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-tmp-dir\") pod \"bd572114-bd0e-4dbc-a04b-5cd05336e333\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " Jul 16 18:24:39.531461 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531415 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd572114-bd0e-4dbc-a04b-5cd05336e333-tls-certs\") pod \"bd572114-bd0e-4dbc-a04b-5cd05336e333\" (UID: \"bd572114-bd0e-4dbc-a04b-5cd05336e333\") " Jul 16 18:24:39.531654 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531454 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-home" (OuterVolumeSpecName: "home") pod "bd572114-bd0e-4dbc-a04b-5cd05336e333" (UID: "bd572114-bd0e-4dbc-a04b-5cd05336e333"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:24:39.531654 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531488 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd572114-bd0e-4dbc-a04b-5cd05336e333" (UID: "bd572114-bd0e-4dbc-a04b-5cd05336e333"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:24:39.531654 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531633 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd572114-bd0e-4dbc-a04b-5cd05336e333" (UID: "bd572114-bd0e-4dbc-a04b-5cd05336e333"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:24:39.531826 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531778 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-home\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:39.531826 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531794 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-model-cache\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:39.531826 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.531804 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-tmp-dir\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:39.533280 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.533253 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-dshm" (OuterVolumeSpecName: "dshm") pod "bd572114-bd0e-4dbc-a04b-5cd05336e333" (UID: "bd572114-bd0e-4dbc-a04b-5cd05336e333"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:24:39.533478 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.533458 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd572114-bd0e-4dbc-a04b-5cd05336e333-kube-api-access-97d62" (OuterVolumeSpecName: "kube-api-access-97d62") pod "bd572114-bd0e-4dbc-a04b-5cd05336e333" (UID: "bd572114-bd0e-4dbc-a04b-5cd05336e333"). InnerVolumeSpecName "kube-api-access-97d62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:24:39.533575 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.533556 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd572114-bd0e-4dbc-a04b-5cd05336e333-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd572114-bd0e-4dbc-a04b-5cd05336e333" (UID: "bd572114-bd0e-4dbc-a04b-5cd05336e333"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:24:39.632554 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.632518 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd572114-bd0e-4dbc-a04b-5cd05336e333-tls-certs\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:39.632554 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.632548 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd572114-bd0e-4dbc-a04b-5cd05336e333-dshm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:39.632554 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:39.632558 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97d62\" (UniqueName: \"kubernetes.io/projected/bd572114-bd0e-4dbc-a04b-5cd05336e333-kube-api-access-97d62\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:40.013846 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.013809 2576 generic.go:356] "Generic (PLEG): container finished" podID="bd572114-bd0e-4dbc-a04b-5cd05336e333" containerID="54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6" exitCode=0 Jul 16 18:24:40.014005 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.013860 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" Jul 16 18:24:40.014005 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.013899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" event={"ID":"bd572114-bd0e-4dbc-a04b-5cd05336e333","Type":"ContainerDied","Data":"54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6"} Jul 16 18:24:40.014005 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.013946 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5" event={"ID":"bd572114-bd0e-4dbc-a04b-5cd05336e333","Type":"ContainerDied","Data":"ac1ef92b500f3fe6db00f01dc3dbff1929a5faf7ec918e474f7a009c049a2a8f"} Jul 16 18:24:40.014005 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.013965 2576 scope.go:117] "RemoveContainer" containerID="54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6" Jul 16 18:24:40.023317 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.023292 2576 scope.go:117] "RemoveContainer" containerID="54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6" Jul 16 18:24:40.023578 ip-10-0-128-217 kubenswrapper[2576]: E0716 18:24:40.023550 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6\": container with ID starting with 54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6 not found: ID does not exist" containerID="54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6" Jul 16 18:24:40.023633 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.023587 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6"} err="failed to get container status \"54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6\": rpc error: code = NotFound desc = could not find container \"54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6\": container with ID starting with 54b7913ffcc768480254005fcbe49b3d41e5db7d3c805dc74c31b2919cd917c6 not found: ID does not exist" Jul 16 18:24:40.038054 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.038028 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5"] Jul 16 18:24:40.042039 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.042014 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a0389e/rate-test-model-a0389e-kserve-678595c595-xk5s5"] Jul 16 18:24:40.096922 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.096869 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" podUID="a9ada32a-e4f0-4f85-9f82-00bc1e80b034" containerName="main" containerID="cri-o://f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197" gracePeriod=45 Jul 16 18:24:40.351987 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.351965 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:24:40.441783 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.441750 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b82r\" (UniqueName: \"kubernetes.io/projected/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-kube-api-access-8b82r\") pod \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " Jul 16 18:24:40.441919 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.441815 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tmp-dir\") pod \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " Jul 16 18:24:40.441919 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.441838 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-dshm\") pod \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " Jul 16 18:24:40.442005 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.441941 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-model-cache\") pod \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " Jul 16 18:24:40.442094 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.442078 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-home\") pod \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " Jul 16 18:24:40.442142 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.442125 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tls-certs\") pod \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\" (UID: \"a9ada32a-e4f0-4f85-9f82-00bc1e80b034\") " Jul 16 18:24:40.442142 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.442079 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9ada32a-e4f0-4f85-9f82-00bc1e80b034" (UID: "a9ada32a-e4f0-4f85-9f82-00bc1e80b034"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:24:40.442229 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.442174 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9ada32a-e4f0-4f85-9f82-00bc1e80b034" (UID: "a9ada32a-e4f0-4f85-9f82-00bc1e80b034"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:24:40.442283 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.442251 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-home" (OuterVolumeSpecName: "home") pod "a9ada32a-e4f0-4f85-9f82-00bc1e80b034" (UID: "a9ada32a-e4f0-4f85-9f82-00bc1e80b034"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:24:40.442434 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.442407 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-home\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:40.442434 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.442430 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tmp-dir\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:40.442434 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.442440 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-model-cache\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:40.443981 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.443957 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-dshm" (OuterVolumeSpecName: "dshm") pod "a9ada32a-e4f0-4f85-9f82-00bc1e80b034" (UID: "a9ada32a-e4f0-4f85-9f82-00bc1e80b034"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:24:40.444104 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.444090 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-kube-api-access-8b82r" (OuterVolumeSpecName: "kube-api-access-8b82r") pod "a9ada32a-e4f0-4f85-9f82-00bc1e80b034" (UID: "a9ada32a-e4f0-4f85-9f82-00bc1e80b034"). InnerVolumeSpecName "kube-api-access-8b82r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:24:40.444175 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.444153 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9ada32a-e4f0-4f85-9f82-00bc1e80b034" (UID: "a9ada32a-e4f0-4f85-9f82-00bc1e80b034"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:24:40.543095 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.543012 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-tls-certs\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:40.543095 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.543039 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b82r\" (UniqueName: \"kubernetes.io/projected/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-kube-api-access-8b82r\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:40.543095 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.543048 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ada32a-e4f0-4f85-9f82-00bc1e80b034-dshm\") on node \"ip-10-0-128-217.ec2.internal\" DevicePath \"\"" Jul 16 18:24:40.771854 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:40.771820 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd572114-bd0e-4dbc-a04b-5cd05336e333" path="/var/lib/kubelet/pods/bd572114-bd0e-4dbc-a04b-5cd05336e333/volumes" Jul 16 18:24:41.021923 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:41.021890 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9ada32a-e4f0-4f85-9f82-00bc1e80b034" containerID="f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197" exitCode=0 Jul 16 18:24:41.022088 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:41.021932 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" Jul 16 18:24:41.022088 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:41.021975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" event={"ID":"a9ada32a-e4f0-4f85-9f82-00bc1e80b034","Type":"ContainerDied","Data":"f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197"} Jul 16 18:24:41.022088 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:41.022016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd" event={"ID":"a9ada32a-e4f0-4f85-9f82-00bc1e80b034","Type":"ContainerDied","Data":"2c5d3616baa859b0626c3ec2bfe9cb7d7efe3cf4ab593fc76bf29761ee281d43"} Jul 16 18:24:41.022088 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:41.022035 2576 scope.go:117] "RemoveContainer" containerID="f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197" Jul 16 18:24:41.030309 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:41.030288 2576 scope.go:117] "RemoveContainer" containerID="f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197" Jul 16 18:24:41.030595 ip-10-0-128-217 kubenswrapper[2576]: E0716 18:24:41.030575 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197\": container with ID starting with f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197 not found: ID does not exist" containerID="f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197" Jul 16 18:24:41.030635 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:41.030603 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197"} err="failed to get container status \"f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197\": rpc error: code = NotFound desc = could not find container \"f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197\": container with ID starting with f521160a8378dd852ecd1c400152c0c079486c98c4af9e3350849759af544197 not found: ID does not exist" Jul 16 18:24:41.039960 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:41.039936 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd"] Jul 16 18:24:41.043760 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:41.043737 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc54de/rate-test-model-bc54de-kserve-55f687cf5-rhbbd"] Jul 16 18:24:42.772001 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:42.771968 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ada32a-e4f0-4f85-9f82-00bc1e80b034" path="/var/lib/kubelet/pods/a9ada32a-e4f0-4f85-9f82-00bc1e80b034/volumes" Jul 16 18:24:57.895581 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:24:57.895537 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:25:20.713695 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:20.713613 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g662c"] Jul 16 18:25:37.433945 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:37.433919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674ff8dd58-jf9t4_0f7a6884-e166-445a-ac18-6fcb34814859/copy-manifests/0.log" Jul 16 18:25:37.443179 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:37.443155 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674ff8dd58-jf9t4_0f7a6884-e166-445a-ac18-6fcb34814859/manager/0.log" Jul 16 18:25:42.173148 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:42.173116 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g662c_312b41b6-d2e8-4b5b-910f-16eff440ff58/limitador/0.log" Jul 16 18:25:43.055353 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:43.055319 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5966557cf8-xn9sc_69be8998-c243-473c-a48e-26b5f9a5d0bb/kube-auth-proxy/0.log" Jul 16 18:25:44.082242 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:44.082211 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wrw7m_898ea9e4-e232-49e6-acae-0f4dd39ca165/main/0.log" Jul 16 18:25:44.327432 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:44.327402 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdbmwg_d3660f9e-fd0e-4f0d-840a-9006fcb578c2/main/0.log" Jul 16 18:25:44.458713 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:44.458681 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-sw57n_2863783a-f9cd-4381-9865-e697d3d85bca/main/0.log" Jul 16 18:25:44.583547 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:44.583520 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-d2pmd_65a81e21-e810-469b-af2d-6a5c4636dbf9/main/0.log" Jul 16 18:25:54.291265 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:54.291236 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-65qpf_cb2e0f7c-1598-4d6f-8515-07473e51b8c9/global-pull-secret-syncer/0.log" Jul 16 18:25:54.698739 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:54.698691 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qzrj4_6e9ae6c1-ca64-4d54-9283-3cc84e8300bf/konnectivity-agent/0.log" Jul 16 18:25:54.738618 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:25:54.738591 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-217.ec2.internal_cc34ace512f7ad807781d2c531ab57e7/haproxy/0.log" Jul 16 18:26:00.292866 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:00.292837 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g662c_312b41b6-d2e8-4b5b-910f-16eff440ff58/limitador/0.log" Jul 16 18:26:02.163052 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.163022 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b8c9990-6949-406b-aa0f-3caad6269a15/alertmanager/0.log" Jul 16 18:26:02.195936 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.195908 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b8c9990-6949-406b-aa0f-3caad6269a15/config-reloader/0.log" Jul 16 18:26:02.236920 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.236891 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b8c9990-6949-406b-aa0f-3caad6269a15/kube-rbac-proxy-web/0.log" Jul 16 18:26:02.265052 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.265009 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b8c9990-6949-406b-aa0f-3caad6269a15/kube-rbac-proxy/0.log" Jul 16 18:26:02.293424 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.293391 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b8c9990-6949-406b-aa0f-3caad6269a15/kube-rbac-proxy-metric/0.log" Jul 16 18:26:02.325875 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.325847 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b8c9990-6949-406b-aa0f-3caad6269a15/prom-label-proxy/0.log" Jul 16 18:26:02.359838 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.359809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b8c9990-6949-406b-aa0f-3caad6269a15/init-config-reloader/0.log" Jul 16 18:26:02.449555 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.449450 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-7zhkn_71c818d1-4c62-4acf-aa66-f0bd065eee9b/kube-state-metrics/0.log" Jul 16 18:26:02.481632 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.481605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-7zhkn_71c818d1-4c62-4acf-aa66-f0bd065eee9b/kube-rbac-proxy-main/0.log" Jul 16 18:26:02.515336 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.515311 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-7zhkn_71c818d1-4c62-4acf-aa66-f0bd065eee9b/kube-rbac-proxy-self/0.log" Jul 16 18:26:02.547631 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.547605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-557cf879c5-tm4hc_edfab048-9ffd-4be9-96f1-0909deb3a0a4/metrics-server/0.log" Jul 16 18:26:02.716333 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.716252 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rlxm_bbdfa7f8-fa05-48c7-9cf3-ac75acff0006/node-exporter/0.log" Jul 16 18:26:02.746446 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.746423 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rlxm_bbdfa7f8-fa05-48c7-9cf3-ac75acff0006/kube-rbac-proxy/0.log" Jul 16 18:26:02.776395 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.776366 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rlxm_bbdfa7f8-fa05-48c7-9cf3-ac75acff0006/init-textfile/0.log" Jul 16 18:26:02.910827 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.910795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-952wv_9c65fe41-57a0-4741-bcd0-3eef39882bde/kube-rbac-proxy-main/0.log" Jul 16 18:26:02.947077 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.947043 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-952wv_9c65fe41-57a0-4741-bcd0-3eef39882bde/kube-rbac-proxy-self/0.log" Jul 16 18:26:02.980067 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:02.979984 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-952wv_9c65fe41-57a0-4741-bcd0-3eef39882bde/openshift-state-metrics/0.log" Jul 16 18:26:03.379303 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:03.379222 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-qvkrm_de07317d-4b69-4ad5-b00a-a0a4d52308ce/prometheus-operator/0.log" Jul 16 18:26:03.407547 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:03.407518 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-qvkrm_de07317d-4b69-4ad5-b00a-a0a4d52308ce/kube-rbac-proxy/0.log" Jul 16 18:26:03.441624 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:03.441595 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-kj5j2_9e1f0cc6-05dd-425c-af5e-b69ea95dae97/prometheus-operator-admission-webhook/0.log" Jul 16 18:26:03.479329 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:03.479306 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54fbbb49cc-x5kfq_f357bf8c-2d12-44d9-a8d3-58c914163edd/telemeter-client/0.log" Jul 16 18:26:03.511556 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:03.511530 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54fbbb49cc-x5kfq_f357bf8c-2d12-44d9-a8d3-58c914163edd/reload/0.log" Jul 16 18:26:03.541339 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:03.541318 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54fbbb49cc-x5kfq_f357bf8c-2d12-44d9-a8d3-58c914163edd/kube-rbac-proxy/0.log" Jul 16 18:26:04.933925 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.933893 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n"] Jul 16 18:26:04.934298 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.934235 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ada32a-e4f0-4f85-9f82-00bc1e80b034" containerName="main" Jul 16 18:26:04.934298 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.934246 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ada32a-e4f0-4f85-9f82-00bc1e80b034" containerName="main" Jul 16 18:26:04.934298 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.934254 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd572114-bd0e-4dbc-a04b-5cd05336e333" containerName="main" Jul 16 18:26:04.934298 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.934260 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd572114-bd0e-4dbc-a04b-5cd05336e333" containerName="main" Jul 16 18:26:04.934430 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.934331 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd572114-bd0e-4dbc-a04b-5cd05336e333" containerName="main" Jul 16 18:26:04.934430 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.934340 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ada32a-e4f0-4f85-9f82-00bc1e80b034" containerName="main" Jul 16 18:26:04.937416 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.937396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:04.941826 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.941807 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p6ppj\"/\"default-dockercfg-46jhl\"" Jul 16 18:26:04.941941 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.941828 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p6ppj\"/\"kube-root-ca.crt\"" Jul 16 18:26:04.942229 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.942213 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p6ppj\"/\"openshift-service-ca.crt\"" Jul 16 18:26:04.958093 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:04.958062 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n"] Jul 16 18:26:05.039072 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.039029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-proc\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.039072 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.039070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-lib-modules\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.039306 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.039135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-podres\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.039306 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.039212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-sys\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.039306 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.039286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rrc\" (UniqueName: \"kubernetes.io/projected/6d645fec-78ba-4fba-b3d0-c2100840274a-kube-api-access-95rrc\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.140596 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.140555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-proc\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.140596 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.140595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-lib-modules\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.140912 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.140624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-podres\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.140912 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.140688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-sys\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.140912 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.140771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rrc\" (UniqueName: \"kubernetes.io/projected/6d645fec-78ba-4fba-b3d0-c2100840274a-kube-api-access-95rrc\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.140912 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.140780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-lib-modules\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.140912 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.140780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-proc\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.140912 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.140837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-sys\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.140912 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.140845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d645fec-78ba-4fba-b3d0-c2100840274a-podres\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.150690 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.150669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rrc\" (UniqueName: \"kubernetes.io/projected/6d645fec-78ba-4fba-b3d0-c2100840274a-kube-api-access-95rrc\") pod \"perf-node-gather-daemonset-sgp7n\" (UID: \"6d645fec-78ba-4fba-b3d0-c2100840274a\") " pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.247463 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.247374 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:05.379993 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.379969 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n"] Jul 16 18:26:05.382138 ip-10-0-128-217 kubenswrapper[2576]: W0716 18:26:05.382110 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6d645fec_78ba_4fba_b3d0_c2100840274a.slice/crio-8c7b75f774cc4e558fa0ce2672272418a45d1d405c7b20588fbe25b459d88a68 WatchSource:0}: Error finding container 8c7b75f774cc4e558fa0ce2672272418a45d1d405c7b20588fbe25b459d88a68: Status 404 returned error can't find the container with id 8c7b75f774cc4e558fa0ce2672272418a45d1d405c7b20588fbe25b459d88a68 Jul 16 18:26:05.420225 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:05.420194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" event={"ID":"6d645fec-78ba-4fba-b3d0-c2100840274a","Type":"ContainerStarted","Data":"8c7b75f774cc4e558fa0ce2672272418a45d1d405c7b20588fbe25b459d88a68"} Jul 16 18:26:06.112562 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:06.112532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-6hbd4_aebf3b4e-aaa5-4156-a625-4c344b053213/download-server/0.log" Jul 16 18:26:06.426825 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:06.426789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" event={"ID":"6d645fec-78ba-4fba-b3d0-c2100840274a","Type":"ContainerStarted","Data":"6501e20a38f2e4d6f154371347c8201b7a9536ef087e8a91ea9d811f72a36781"} Jul 16 18:26:06.427007 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:06.426901 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:06.454482 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:06.454422 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" podStartSLOduration=2.454407525 podStartE2EDuration="2.454407525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:26:06.453168139 +0000 UTC m=+3210.114731773" watchObservedRunningTime="2026-07-16 18:26:06.454407525 +0000 UTC m=+3210.115971158" Jul 16 18:26:08.182944 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:08.182917 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xtq5p_8ce9162f-6d4d-44ec-a1ea-4d5e2652c758/dns/0.log" Jul 16 18:26:08.212184 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:08.212159 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xtq5p_8ce9162f-6d4d-44ec-a1ea-4d5e2652c758/kube-rbac-proxy/0.log" Jul 16 18:26:08.292803 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:08.292770 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dfr8c_4da5bf01-f439-4e51-8ef5-6828fee14726/dns-node-resolver/0.log" Jul 16 18:26:09.019485 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:09.019459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sc2p8_ec0ea45e-427b-42b5-a095-2d6e8a0ef43f/node-ca/0.log" Jul 16 18:26:10.254860 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:10.254829 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5966557cf8-xn9sc_69be8998-c243-473c-a48e-26b5f9a5d0bb/kube-auth-proxy/0.log" Jul 16 18:26:11.163166 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:11.163133 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cbc5s_86b5d1c9-6f4b-4898-8758-b354d8af0c38/serve-healthcheck-canary/0.log" Jul 16 18:26:12.010090 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:12.010056 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-spllq_fd09d19f-aca3-43a5-ba18-82f2b996dba4/kube-rbac-proxy/0.log" Jul 16 18:26:12.042117 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:12.042088 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-spllq_fd09d19f-aca3-43a5-ba18-82f2b996dba4/exporter/0.log" Jul 16 18:26:12.073524 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:12.073496 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-spllq_fd09d19f-aca3-43a5-ba18-82f2b996dba4/extractor/0.log" Jul 16 18:26:12.442026 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:12.442001 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p6ppj/perf-node-gather-daemonset-sgp7n" Jul 16 18:26:14.361640 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:14.361563 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5966557cf8-xn9sc_69be8998-c243-473c-a48e-26b5f9a5d0bb/kube-auth-proxy/0.log" Jul 16 18:26:15.374251 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:15.374225 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674ff8dd58-jf9t4_0f7a6884-e166-445a-ac18-6fcb34814859/manager/0.log" Jul 16 18:26:15.423708 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:15.423683 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674ff8dd58-jf9t4_0f7a6884-e166-445a-ac18-6fcb34814859/copy-manifests/0.log" Jul 16 18:26:15.969848 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:15.969822 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5966557cf8-xn9sc_69be8998-c243-473c-a48e-26b5f9a5d0bb/kube-auth-proxy/0.log" Jul 16 18:26:17.056443 ip-10-0-128-217 kubenswrapper[2576]: I0716 18:26:17.056411 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65794b4fd8-5kkmm_857205c0-47c2-43e4-90ce-d45ff2779866/manager/0.log"