Jul 07 15:39:19.874768 ip-10-0-128-121 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 15:39:19.874779 ip-10-0-128-121 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 15:39:19.874789 ip-10-0-128-121 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 15:39:19.875135 ip-10-0-128-121 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 15:39:30.029893 ip-10-0-128-121 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 15:39:30.029913 ip-10-0-128-121 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 27194ae0e8df4ae9a4d7f58d1fc96ed5 -- Jul 07 15:41:32.232186 ip-10-0-128-121 systemd[1]: Starting Kubernetes Kubelet... Jul 07 15:41:34.385323 ip-10-0-128-121 systemd[1]: Started Kubernetes Kubelet. Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.308297 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313703 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313715 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313719 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313722 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313725 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313728 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313731 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:34.435768 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313735 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313739 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313741 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313744 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313747 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313751 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313755 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313758 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313760 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313763 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313766 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313768 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313771 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313775 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313779 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313782 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313785 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313788 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313791 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313797 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:34.539078 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313799 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313802 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313804 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313807 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313809 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313812 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313815 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313818 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313821 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313825 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313829 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313832 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313835 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313838 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313841 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313843 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313846 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313848 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313851 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313853 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:34.540200 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313856 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313858 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313861 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313864 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313866 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313869 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313871 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313874 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313877 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313880 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313884 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313886 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313889 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313891 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313894 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313896 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313899 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313901 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313904 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:34.541016 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313907 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313909 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313912 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313915 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313918 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313921 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313923 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313926 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313928 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313931 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313935 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313938 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313940 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313943 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313945 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313948 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313950 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313953 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313956 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313958 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:34.542615 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313961 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313963 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313966 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313969 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313971 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313974 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313977 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313979 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313982 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313984 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313988 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313990 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.313993 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314717 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314724 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314727 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314730 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314732 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314735 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314737 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:34.543345 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314740 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314742 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314745 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314748 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314751 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314754 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314756 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314759 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314761 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314764 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314767 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314769 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314772 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314775 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314778 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314781 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314784 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314787 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314789 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314791 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:34.544528 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314794 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314796 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314799 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314802 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314804 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314807 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314809 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314812 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314815 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314819 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314822 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314825 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314827 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314830 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314832 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314835 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314837 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314840 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314842 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:34.545320 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314845 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314848 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314850 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314854 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314856 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314859 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314861 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314864 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314867 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314870 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314872 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314875 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314878 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314880 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314885 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314888 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314891 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314894 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314897 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:34.546492 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314899 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314902 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314905 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314907 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314910 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314912 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314915 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314917 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314919 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314922 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314924 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314927 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314929 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314931 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314934 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314936 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314939 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314941 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314944 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314949 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:34.548477 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314951 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314955 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314957 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314960 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314963 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314966 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314968 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314971 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314973 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314976 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314978 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314982 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314985 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.314988 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315380 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315385 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315388 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315392 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315395 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:34.549024 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315399 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315402 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315405 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315407 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315410 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315413 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315415 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315418 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315420 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315423 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315425 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315429 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315433 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315435 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315438 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315441 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315443 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315446 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315451 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:34.549616 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315453 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315456 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315458 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315461 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315463 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315466 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315468 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315470 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315473 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315476 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315478 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315481 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315483 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315486 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315489 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315491 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315493 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315496 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315498 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315501 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:34.550063 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315503 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315506 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315508 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315510 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315513 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315517 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315520 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315522 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315525 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315527 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315530 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315533 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315536 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315538 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315560 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315563 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315566 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315568 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315571 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315574 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:34.550536 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315577 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315579 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315582 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315585 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315588 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315591 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315593 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315596 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315598 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315601 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315603 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315606 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315611 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315614 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315617 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315620 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315622 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315627 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315630 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315633 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:34.551029 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315636 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315639 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315641 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315645 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315648 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315650 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315654 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315657 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315660 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315662 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315665 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315668 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315670 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315673 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.315675 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315739 2583 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315751 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315758 2583 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315762 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315766 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315769 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 15:41:34.551496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315774 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315778 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315781 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315784 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315788 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315791 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315794 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315797 2583 flags.go:64] FLAG: --cgroup-root="" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315802 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315805 2583 flags.go:64] FLAG: --client-ca-file="" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315809 2583 flags.go:64] FLAG: --cloud-provider="external" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315812 2583 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315816 2583 flags.go:64] FLAG: --cluster-domain="" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315819 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315823 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315826 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315829 2583 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315833 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315836 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315839 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315842 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315846 2583 flags.go:64] FLAG: --contention-profiling="false" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315848 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315851 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 15:41:34.552003 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315854 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315857 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315861 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315864 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315867 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315869 2583 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315872 2583 flags.go:64] FLAG: --enable-server="true" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315875 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315879 2583 flags.go:64] FLAG: --event-burst="100" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315882 2583 flags.go:64] FLAG: --event-qps="50" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315885 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315888 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315891 2583 flags.go:64] FLAG: --eviction-hard="" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315894 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315897 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315900 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315905 2583 flags.go:64] FLAG: --eviction-soft="" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315908 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315911 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315915 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315918 2583 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315920 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315925 2583 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315927 2583 flags.go:64] FLAG: --feature-gates="" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315931 2583 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 15:41:34.552564 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315934 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315937 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315940 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315943 2583 flags.go:64] FLAG: --healthz-port="10248" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315946 2583 flags.go:64] FLAG: --help="false" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315949 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315952 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315955 2583 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315958 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315961 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315965 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315967 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315970 2583 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315973 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315976 2583 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315978 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315982 2583 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315984 2583 flags.go:64] FLAG: --kube-reserved="" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315987 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315990 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315993 2583 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315995 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.315998 2583 flags.go:64] FLAG: --lock-file="" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316002 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 15:41:34.553199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316005 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316008 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316013 2583 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316017 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316019 2583 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316023 2583 flags.go:64] FLAG: --logging-format="text" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316026 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316030 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316032 2583 flags.go:64] FLAG: --manifest-url="" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316035 2583 flags.go:64] FLAG: --manifest-url-header="" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316039 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316042 2583 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316046 2583 flags.go:64] FLAG: --max-pods="110" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316049 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316052 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316055 2583 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316058 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316061 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316064 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316066 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316073 2583 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316076 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316079 2583 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 15:41:34.553779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316082 2583 flags.go:64] FLAG: --pod-cidr="" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316085 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316091 2583 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316094 2583 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316097 2583 flags.go:64] FLAG: --pods-per-core="0" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316100 2583 flags.go:64] FLAG: --port="10250" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316103 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316106 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0780ebd4ce1762d79" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316111 2583 flags.go:64] FLAG: --qos-reserved="" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316114 2583 flags.go:64] FLAG: --read-only-port="10255" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316117 2583 flags.go:64] FLAG: --register-node="true" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316120 2583 flags.go:64] FLAG: --register-with-taints="" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316123 2583 flags.go:64] FLAG: --registry-burst="10" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316127 2583 flags.go:64] FLAG: --registry-qps="5" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316130 2583 flags.go:64] FLAG: --reserved-cpus="" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316134 2583 flags.go:64] FLAG: --reserved-memory="" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316137 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316140 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316143 2583 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316146 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316149 2583 flags.go:64] FLAG: --runonce="false" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316151 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316154 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316157 2583 flags.go:64] FLAG: --seccomp-default="false" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316160 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 15:41:34.554319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316163 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316166 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316169 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316172 2583 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316175 2583 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316177 2583 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316180 2583 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316183 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316186 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316188 2583 flags.go:64] FLAG: --system-cgroups="" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316191 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316196 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316199 2583 flags.go:64] FLAG: --tls-cert-file="" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316201 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316206 2583 flags.go:64] FLAG: --tls-min-version="" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316211 2583 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316213 2583 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316216 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316219 2583 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316222 2583 flags.go:64] FLAG: --v="2" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316226 2583 flags.go:64] FLAG: --version="false" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316230 2583 flags.go:64] FLAG: --vmodule="" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316237 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316240 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316357 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:34.554938 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316361 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316364 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316367 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316370 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316373 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316375 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316378 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316380 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316383 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316385 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316388 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316391 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316393 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316396 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316398 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316401 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316403 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316406 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316409 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316411 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:34.555526 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316413 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316416 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316420 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316423 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316425 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316428 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316430 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316433 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316435 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316439 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316442 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316445 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316447 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316449 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316452 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316455 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316457 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316460 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316462 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316464 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:34.556022 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316467 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316469 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316472 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316475 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316477 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316480 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316482 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316486 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316488 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316491 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316493 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316497 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316500 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316503 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316508 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316510 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316513 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316516 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316518 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:34.556485 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316521 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316524 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316528 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316532 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316535 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316538 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316540 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316561 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316564 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316567 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316570 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316572 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316575 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316577 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316580 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316582 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316585 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316587 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316590 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316593 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:34.556997 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316595 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316598 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316602 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316605 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316608 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316611 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316614 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316617 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316620 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316623 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316626 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316628 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316631 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316633 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316637 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316640 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316643 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316646 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:34.557469 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.316649 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.316655 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.323904 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.323918 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.323931 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.323936 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324002 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324008 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324012 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324016 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324019 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324022 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324024 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324027 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:34.557908 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324030 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324033 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324035 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324038 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324040 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324043 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324046 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324048 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324051 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324054 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324056 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324059 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324062 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324064 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324067 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324070 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324072 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324076 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324079 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324082 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:34.558249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324084 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324090 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324093 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324096 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324099 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324101 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324104 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324106 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324109 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324111 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324114 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324116 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324119 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324121 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324124 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324126 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324129 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324131 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324134 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324136 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:34.558756 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324139 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324141 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324144 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324146 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324150 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324154 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324157 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324160 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324162 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324165 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324168 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324171 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324174 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324177 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324183 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324185 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324188 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324190 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324193 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324196 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:34.559224 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324198 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324201 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324204 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324206 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324209 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324211 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324213 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324216 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324218 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324221 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324223 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324226 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324228 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324231 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324234 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324236 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324239 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324241 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324244 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324246 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:34.559718 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324248 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324251 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324256 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324259 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324261 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324264 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324266 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324274 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324277 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324280 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324284 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.324290 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324425 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324430 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:34.560192 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324433 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324436 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324439 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324442 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324445 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324448 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324450 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324453 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324456 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324460 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324463 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324465 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324468 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324470 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324473 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324475 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324478 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324481 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324483 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324486 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:34.560527 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324488 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324493 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324496 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324498 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324501 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324503 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324506 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324509 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324511 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324514 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324516 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324519 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324521 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324524 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324526 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324529 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324531 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324534 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324536 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324539 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:34.561003 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324542 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324565 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324568 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324570 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324573 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324576 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324578 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324581 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324584 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324586 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324589 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324591 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324594 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324598 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324601 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324603 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324606 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324609 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324611 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324615 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:34.561464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324618 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324621 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324623 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324626 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324628 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324630 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324634 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324637 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324639 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324642 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324644 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324647 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324650 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324653 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324656 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324659 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324662 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324666 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324669 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:34.561986 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324672 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324675 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324677 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324680 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324683 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324686 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324689 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324692 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324695 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324698 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324700 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324703 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324705 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324708 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324711 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324713 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324716 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:34.562446 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:34.324718 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.324723 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.324831 2583 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.327025 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.328477 2583 server.go:1019] "Starting client certificate rotation" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.328592 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.328656 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.331321 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.342698 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.354480 2583 log.go:25] "Validated CRI v1 runtime API" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.356850 2583 log.go:25] "Validated CRI v1 image API" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.357647 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 15:41:34.562873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.360122 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8efcb5ba-cf84-4097-9326-242666489a40:/dev/nvme0n1p3 d43e70b1-f5d0-47b4-a893-6f7b5c772af6:/dev/nvme0n1p4] Jul 07 15:41:34.563164 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.360138 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 15:41:34.563194 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.364151 2583 manager.go:216] Machine: {Timestamp:2026-07-07 15:41:34.363681656 +0000 UTC m=+0.494094793 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102703 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f0880c473b8cd4ab1215a0ae77148 SystemUUID:ec2f0880-c473-b8cd-4ab1-215a0ae77148 BootID:27194ae0-e8df-4ae9-a4d7-f58d1fc96ed5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fe:c0:64:a9:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fe:c0:64:a9:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:fa:c9:5c:3d:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 15:41:34.563194 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.364242 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 15:41:34.563194 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.364311 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 15:41:34.563194 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.364596 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 15:41:34.563194 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.366843 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 15:41:34.563194 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.366869 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 15:41:34.563194 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.368242 2583 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.368251 2583 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.368259 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.369241 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.370964 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.374192 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.374309 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.374338 2583 kubelet.go:486] "Attempting to sync node with API server" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.374347 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.374369 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.374380 2583 kubelet.go:398] "Adding apiserver pod source" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.374391 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.374741 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.374764 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.376840 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.379284 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.379313 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381274 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381292 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381299 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381307 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381316 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381323 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381332 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381339 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381359 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381366 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381384 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381390 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381409 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.381414 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.384035 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.384182 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.384363 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.384469 2583 server.go:1268] "Started kubelet" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.385068 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.388600 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.388648 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.388764 2583 server.go:310] "Adding debug handlers to kubelet server" Jul 07 15:41:34.563393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.390234 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.391865 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.397194 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.397495 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.398295 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.398321 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.399073 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.399084 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.399173 2583 reconstruct.go:97] "Volume reconstruction finished" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.399178 2583 reconciler.go:29] "Reconciler: start to sync state" Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.401597 2583 factory.go:55] Registering systemd factory Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.401613 2583 factory.go:223] Registration of the systemd container factory successfully Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.401811 2583 factory.go:153] Registering CRI-O factory Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.401824 2583 factory.go:223] Registration of the crio container factory successfully Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.401875 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.401905 2583 factory.go:103] Registering Raw factory Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.401917 2583 manager.go:1196] Started watching for new ooms in manager Jul 07 15:41:34.564332 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.498982 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:34.595891 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.595867 2583 manager.go:349] Starting recovery of all containers Jul 07 15:41:34.596163 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.596136 2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 15:41:34.596247 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.596189 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-smxb9" Jul 07 15:41:34.596247 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.596207 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-smxb9" Jul 07 15:41:34.596417 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.596396 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 15:41:34.597012 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.596041 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-121.ec2.internal.18c00c39b2667da0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-121.ec2.internal,UID:ip-10-0-128-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-121.ec2.internal,},FirstTimestamp:2026-07-07 15:41:34.38444688 +0000 UTC m=+0.514860017,LastTimestamp:2026-07-07 15:41:34.38444688 +0000 UTC m=+0.514860017,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-121.ec2.internal,}" Jul 07 15:41:34.599286 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.599262 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:34.605946 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.605929 2583 manager.go:354] Recovery completed Jul 07 15:41:34.609829 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.609818 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:34.611786 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.611768 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:34.611856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.611799 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:34.611856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.611809 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:34.612381 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.612367 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 15:41:34.612381 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.612377 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:34.612381 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.612380 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 15:41:34.612508 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.612400 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 07 15:41:34.613102 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.613083 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:34.613160 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.613109 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:34.613160 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.613118 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:34.613742 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.613730 2583 policy_none.go:49] "None policy: Start" Jul 07 15:41:34.613779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.613750 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 15:41:34.613779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.613764 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 15:41:34.614127 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.614056 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-121.ec2.internal.18c00c39bff36772 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-121.ec2.internal,UID:ip-10-0-128-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-121.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-121.ec2.internal,},FirstTimestamp:2026-07-07 15:41:34.611785586 +0000 UTC m=+0.742198722,LastTimestamp:2026-07-07 15:41:34.611785586 +0000 UTC m=+0.742198722,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-121.ec2.internal,}" Jul 07 15:41:34.614994 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.614982 2583 policy_none.go:47] "Start" Jul 07 15:41:34.657198 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.657070 2583 manager.go:337] "Starting Device Plugin manager" Jul 07 15:41:34.657277 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.657224 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 15:41:34.657277 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.657238 2583 server.go:87] "Starting device plugin registration server" Jul 07 15:41:34.657425 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.657415 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 15:41:34.657480 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.657426 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 15:41:34.657559 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.657528 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 15:41:34.657635 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.657623 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 15:41:34.657690 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.657637 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 15:41:34.658078 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.658060 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 15:41:34.658147 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.658101 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:34.758045 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.758013 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:34.759157 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.758959 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:34.759157 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.758988 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:34.759157 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.758999 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:34.759157 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.759023 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.768289 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.768269 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.768377 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.768295 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-121.ec2.internal\": node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:34.783499 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.783482 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:34.790479 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.790436 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 15:41:34.791635 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.791612 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 15:41:34.791711 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.791655 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 15:41:34.791711 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.791677 2583 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 15:41:34.791804 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.791715 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 15:41:34.793750 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.793730 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:34.884590 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.884542 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:34.892699 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.892677 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal"] Jul 07 15:41:34.892754 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.892745 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:34.893661 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.893645 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:34.893739 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.893678 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:34.893739 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.893694 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:34.895333 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.895318 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:34.895464 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.895436 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.895513 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.895466 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:34.901831 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.901814 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3476c52a265caaabfc184395d47554a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"3476c52a265caaabfc184395d47554a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.901914 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.901837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3476c52a265caaabfc184395d47554a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"3476c52a265caaabfc184395d47554a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.901914 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.901856 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dbf149aa0654ab8e90cfec7925c91095-config\") pod \"kube-apiserver-proxy-ip-10-0-128-121.ec2.internal\" (UID: \"dbf149aa0654ab8e90cfec7925c91095\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.901914 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.901905 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:34.902061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.901937 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:34.902061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.901944 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:34.902061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.901952 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:34.902061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.901967 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:34.902061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.901980 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:34.903354 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.903338 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.903433 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.903367 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:34.904138 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.904123 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:34.904230 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.904147 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:34.904230 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:34.904156 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:34.933087 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.933068 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-121.ec2.internal\" not found" node="ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.937258 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.937245 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-121.ec2.internal\" not found" node="ip-10-0-128-121.ec2.internal" Jul 07 15:41:34.985350 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:34.985328 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.002555 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.002526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3476c52a265caaabfc184395d47554a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"3476c52a265caaabfc184395d47554a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" Jul 07 15:41:35.002624 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.002562 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dbf149aa0654ab8e90cfec7925c91095-config\") pod \"kube-apiserver-proxy-ip-10-0-128-121.ec2.internal\" (UID: \"dbf149aa0654ab8e90cfec7925c91095\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" Jul 07 15:41:35.002624 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.002578 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3476c52a265caaabfc184395d47554a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"3476c52a265caaabfc184395d47554a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" Jul 07 15:41:35.002699 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.002623 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dbf149aa0654ab8e90cfec7925c91095-config\") pod \"kube-apiserver-proxy-ip-10-0-128-121.ec2.internal\" (UID: \"dbf149aa0654ab8e90cfec7925c91095\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" Jul 07 15:41:35.002699 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.002631 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3476c52a265caaabfc184395d47554a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"3476c52a265caaabfc184395d47554a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" Jul 07 15:41:35.002699 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.002678 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3476c52a265caaabfc184395d47554a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"3476c52a265caaabfc184395d47554a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" Jul 07 15:41:35.086164 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:35.086104 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.186611 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:35.186583 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.237050 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.237025 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" Jul 07 15:41:35.241067 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.240934 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" Jul 07 15:41:35.287295 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:35.287269 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.327728 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.327706 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 15:41:35.327842 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.327824 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 15:41:35.387835 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:35.387779 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.488724 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:35.488680 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.544892 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.544867 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:35.589559 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:35.589524 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.600828 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.600803 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 15:36:34 +0000 UTC" deadline="2028-03-31 10:47:17.23719329 +0000 UTC" Jul 07 15:41:35.600828 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.600823 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15187h5m41.636372251s" Jul 07 15:41:35.690301 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:35.690237 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.790683 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:35.790656 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.876984 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:35.876951 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf149aa0654ab8e90cfec7925c91095.slice/crio-7ec035e8c0e527647bf0f10688ea8e7c47b53fd2301ca496038bd28a8ac81aaf WatchSource:0}: Error finding container 7ec035e8c0e527647bf0f10688ea8e7c47b53fd2301ca496038bd28a8ac81aaf: Status 404 returned error can't find the container with id 7ec035e8c0e527647bf0f10688ea8e7c47b53fd2301ca496038bd28a8ac81aaf Jul 07 15:41:35.877225 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:35.877211 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3476c52a265caaabfc184395d47554a7.slice/crio-faa49dc7d41492e1fa7c72aaf79f74539f58f8d6855d60bbf7d99071e7cc0aa7 WatchSource:0}: Error finding container faa49dc7d41492e1fa7c72aaf79f74539f58f8d6855d60bbf7d99071e7cc0aa7: Status 404 returned error can't find the container with id faa49dc7d41492e1fa7c72aaf79f74539f58f8d6855d60bbf7d99071e7cc0aa7 Jul 07 15:41:35.884306 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.884289 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:41:35.891292 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:35.891270 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found" Jul 07 15:41:35.972186 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.972132 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:35.999293 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:35.999273 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" Jul 07 15:41:36.011775 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.011758 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 15:41:36.014008 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.013996 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" Jul 07 15:41:36.015646 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.015634 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:36.032349 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.032334 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 15:41:36.375561 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.375466 2583 apiserver.go:52] "Watching apiserver" Jul 07 15:41:36.385391 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.385359 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 15:41:36.386504 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.386483 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mpk6m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg","openshift-cluster-node-tuning-operator/tuned-tthl2","openshift-image-registry/node-ca-vd7wn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal","openshift-multus/multus-additional-cni-plugins-z2fx4","openshift-network-diagnostics/network-check-target-xkxc8","openshift-ovn-kubernetes/ovnkube-node-wzqpt","kube-system/konnectivity-agent-s86zr","kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal","openshift-dns/node-resolver-psmv2","openshift-multus/multus-tbs66","openshift-multus/network-metrics-daemon-w4chw","openshift-network-operator/iptables-alerter-sqsvb"] Jul 07 15:41:36.388706 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.388692 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.389915 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.389895 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.391423 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.391404 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.392375 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.392356 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.393942 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.393927 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:36.394033 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.394002 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:36.395191 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.395175 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 15:41:36.395265 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.395201 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:41:36.395322 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.395312 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fsrlh\"" Jul 07 15:41:36.395451 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.395432 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 15:41:36.395620 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.395605 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.397136 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.397115 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.398291 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.398276 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:36.398378 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.398340 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:36.399047 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.399032 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 15:41:36.399769 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.399752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:36.401226 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.401198 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:36.401305 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.401277 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:36.403203 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.403182 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.404346 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.404327 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 15:41:36.404656 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.404636 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:41:36.404802 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.404671 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w2mgr\"" Jul 07 15:41:36.404802 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.404681 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 15:41:36.404933 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.404839 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.405349 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.405334 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g9mmh\"" Jul 07 15:41:36.409331 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.409278 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-multus-socket-dir-parent\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.409673 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.409397 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-var-lib-cni-bin\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.409673 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.409456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b719a40-3782-4d09-9dd2-637375e15be2-multus-daemon-config\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.409673 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.409647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-etc-kubernetes\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.409990 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.409684 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-kubelet\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.409990 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.409741 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzk8\" (UniqueName: \"kubernetes.io/projected/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-kube-api-access-xrzk8\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:36.410641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410130 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-ovnkube-config\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.410641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410338 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-sysctl-conf\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.410641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410409 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd55f852-d386-432b-a490-583cc2a4db3d-hosts-file\") pod \"node-resolver-psmv2\" (UID: \"cd55f852-d386-432b-a490-583cc2a4db3d\") " pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.410641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410468 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbsn\" (UniqueName: \"kubernetes.io/projected/cd55f852-d386-432b-a490-583cc2a4db3d-kube-api-access-6mbsn\") pod \"node-resolver-psmv2\" (UID: \"cd55f852-d386-432b-a490-583cc2a4db3d\") " pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.410641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410515 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-cni-bin\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.410641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410570 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-run\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.410641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-sys\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.411130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-env-overrides\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.411130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410703 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7854f1f9-a33b-40b4-b4a5-cea927feb277-host-slash\") pod \"iptables-alerter-sqsvb\" (UID: \"7854f1f9-a33b-40b4-b4a5-cea927feb277\") " pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.411130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410883 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-modprobe-d\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.411130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410920 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-var-lib-openvswitch\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.411130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.410986 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-node-log\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.411130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.411130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-lib-modules\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.411130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411076 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 15:41:36.411130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411116 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-run-k8s-cni-cncf-io\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.411521 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411176 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 15:41:36.411521 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411178 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-etc-openvswitch\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.411521 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411209 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/185cd267-f8e8-4e94-a620-9b5ac374fe19-serviceca\") pod \"node-ca-vd7wn\" (UID: \"185cd267-f8e8-4e94-a620-9b5ac374fe19\") " pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.411521 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-run-netns\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.411521 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411282 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dw24\" (UniqueName: \"kubernetes.io/projected/9b719a40-3782-4d09-9dd2-637375e15be2-kube-api-access-5dw24\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.411521 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411312 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-log-socket\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.411521 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411415 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd55f852-d386-432b-a490-583cc2a4db3d-tmp-dir\") pod \"node-resolver-psmv2\" (UID: \"cd55f852-d386-432b-a490-583cc2a4db3d\") " pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.411865 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411537 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08452baf-e640-4628-8782-25b1cdb96215-agent-certs\") pod \"konnectivity-agent-s86zr\" (UID: \"08452baf-e640-4628-8782-25b1cdb96215\") " pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:36.411865 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411592 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-cni-netd\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.411865 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-ovnkube-script-lib\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.411865 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411659 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/185cd267-f8e8-4e94-a620-9b5ac374fe19-host\") pod \"node-ca-vd7wn\" (UID: \"185cd267-f8e8-4e94-a620-9b5ac374fe19\") " pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.411865 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411689 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-run-netns\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.411865 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411730 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-host\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.411865 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411346 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kcq7p\"" Jul 07 15:41:36.412188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411884 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b719a40-3782-4d09-9dd2-637375e15be2-cni-binary-copy\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.412188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411914 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-var-lib-kubelet\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.412188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.411993 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63087682-353b-454d-9cea-58c6e6a30613-tmp\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.412188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.412007 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c9ppb\"" Jul 07 15:41:36.412188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.412035 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-var-lib-kubelet\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.412188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.412080 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-hostroot\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.412188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.412115 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-multus-conf-dir\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.412188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.412177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-run-ovn\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.412621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.412230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.412719 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.412697 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:36.412793 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.412756 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-sysconfig\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.412883 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-run-multus-certs\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413005 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgpx\" (UniqueName: \"kubernetes.io/projected/7854f1f9-a33b-40b4-b4a5-cea927feb277-kube-api-access-9pgpx\") pod \"iptables-alerter-sqsvb\" (UID: \"7854f1f9-a33b-40b4-b4a5-cea927feb277\") " pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-kubernetes\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413150 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-multus-cni-dir\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413187 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-ovn-node-metrics-cert\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413236 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-system-cni-dir\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413279 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08452baf-e640-4628-8782-25b1cdb96215-konnectivity-ca\") pod \"konnectivity-agent-s86zr\" (UID: \"08452baf-e640-4628-8782-25b1cdb96215\") " pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbfnh\" (UniqueName: \"kubernetes.io/projected/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-kube-api-access-jbfnh\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413708 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57v6\" (UniqueName: \"kubernetes.io/projected/185cd267-f8e8-4e94-a620-9b5ac374fe19-kube-api-access-w57v6\") pod \"node-ca-vd7wn\" (UID: \"185cd267-f8e8-4e94-a620-9b5ac374fe19\") " pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413737 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413834 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-var-lib-cni-multus\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413847 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 15:41:36.414641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-slash\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-run-openvswitch\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413930 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7854f1f9-a33b-40b4-b4a5-cea927feb277-iptables-alerter-script\") pod \"iptables-alerter-sqsvb\" (UID: \"7854f1f9-a33b-40b4-b4a5-cea927feb277\") " pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413937 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413972 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-systemd\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.413999 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63087682-353b-454d-9cea-58c6e6a30613-etc-tuned\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.414040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-os-release\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.414070 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wnr\" (UniqueName: \"kubernetes.io/projected/63087682-353b-454d-9cea-58c6e6a30613-kube-api-access-47wnr\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.414151 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-cnibin\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.414169 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gl5qw\"" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.414201 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-systemd-units\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.414254 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-run-systemd\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.414293 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-sysctl-d\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.415326 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.415434 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ngnsj\"" Jul 07 15:41:36.415580 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.415532 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 15:41:36.416264 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.415535 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n9wxm\"" Jul 07 15:41:36.416264 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.415816 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 15:41:36.416264 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.415854 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 15:41:36.416264 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.415888 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 15:41:36.416264 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.415941 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 15:41:36.416264 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.416015 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 15:41:36.417593 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.417574 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 15:41:36.417745 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.417574 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 15:41:36.417745 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.417624 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9phb\"" Jul 07 15:41:36.433038 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.433017 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 15:41:36.435899 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.435882 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 15:41:36.436237 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.436218 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 15:41:36.438854 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.438823 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 15:41:36.449412 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.449395 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 15:41:36.451015 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.450998 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 15:41:36.458764 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.458746 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 15:41:36.499004 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.498981 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 15:41:36.504353 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.504332 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z5gkm" Jul 07 15:41:36.514197 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.514178 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z5gkm" Jul 07 15:41:36.515302 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-log-socket\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.515386 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515315 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd55f852-d386-432b-a490-583cc2a4db3d-tmp-dir\") pod \"node-resolver-psmv2\" (UID: \"cd55f852-d386-432b-a490-583cc2a4db3d\") " pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.515386 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515342 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08452baf-e640-4628-8782-25b1cdb96215-agent-certs\") pod \"konnectivity-agent-s86zr\" (UID: \"08452baf-e640-4628-8782-25b1cdb96215\") " pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:36.515494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515400 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-log-socket\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.515494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515452 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-cni-netd\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.515494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-ovnkube-script-lib\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.515668 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515646 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-cni-netd\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.515723 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515680 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd55f852-d386-432b-a490-583cc2a4db3d-tmp-dir\") pod \"node-resolver-psmv2\" (UID: \"cd55f852-d386-432b-a490-583cc2a4db3d\") " pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.515723 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515537 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/185cd267-f8e8-4e94-a620-9b5ac374fe19-host\") pod \"node-ca-vd7wn\" (UID: \"185cd267-f8e8-4e94-a620-9b5ac374fe19\") " pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.515816 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35748986-3b58-45b7-a1e0-a77c803c1ad3-dbus\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:36.515816 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515773 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/185cd267-f8e8-4e94-a620-9b5ac374fe19-host\") pod \"node-ca-vd7wn\" (UID: \"185cd267-f8e8-4e94-a620-9b5ac374fe19\") " pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.515816 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515794 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-system-cni-dir\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.515816 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515777 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 15:41:36.516007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515825 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-run-netns\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.516007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515848 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-host\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.516007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515869 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b719a40-3782-4d09-9dd2-637375e15be2-cni-binary-copy\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.516007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515902 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-run-netns\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.516007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-var-lib-kubelet\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.516007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515932 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-host\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.516007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63087682-353b-454d-9cea-58c6e6a30613-tmp\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.516007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515960 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-var-lib-kubelet\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.516007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.515975 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-var-lib-kubelet\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516022 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-hostroot\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-multus-conf-dir\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-run-ovn\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516087 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516135 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-run-ovn\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516137 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-hostroot\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516165 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516183 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-multus-conf-dir\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516224 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516240 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-sysconfig\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516275 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-run-multus-certs\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516298 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-var-lib-kubelet\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.516414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516306 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgpx\" (UniqueName: \"kubernetes.io/projected/7854f1f9-a33b-40b4-b4a5-cea927feb277-kube-api-access-9pgpx\") pod \"iptables-alerter-sqsvb\" (UID: \"7854f1f9-a33b-40b4-b4a5-cea927feb277\") " pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516335 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516342 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-sysconfig\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516390 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-run-multus-certs\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516422 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8mh5\" (UniqueName: \"kubernetes.io/projected/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-kube-api-access-t8mh5\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-kubernetes\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516715 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-multus-cni-dir\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-ovn-node-metrics-cert\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516815 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-ovnkube-script-lib\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516939 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-multus-cni-dir\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516945 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516961 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-kubernetes\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.517040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.516979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-system-cni-dir\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517047 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08452baf-e640-4628-8782-25b1cdb96215-konnectivity-ca\") pod \"konnectivity-agent-s86zr\" (UID: \"08452baf-e640-4628-8782-25b1cdb96215\") " pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517081 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbfnh\" (UniqueName: \"kubernetes.io/projected/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-kube-api-access-jbfnh\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517131 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b719a40-3782-4d09-9dd2-637375e15be2-cni-binary-copy\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57v6\" (UniqueName: \"kubernetes.io/projected/185cd267-f8e8-4e94-a620-9b5ac374fe19-kube-api-access-w57v6\") pod \"node-ca-vd7wn\" (UID: \"185cd267-f8e8-4e94-a620-9b5ac374fe19\") " pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517174 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517229 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-sys-fs\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-var-lib-cni-multus\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517307 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-system-cni-dir\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517367 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-slash\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517400 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-run-openvswitch\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7854f1f9-a33b-40b4-b4a5-cea927feb277-iptables-alerter-script\") pod \"iptables-alerter-sqsvb\" (UID: \"7854f1f9-a33b-40b4-b4a5-cea927feb277\") " pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517467 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-cnibin\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.517621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517562 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517633 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08452baf-e640-4628-8782-25b1cdb96215-konnectivity-ca\") pod \"konnectivity-agent-s86zr\" (UID: \"08452baf-e640-4628-8782-25b1cdb96215\") " pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517687 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-slash\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517685 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-var-lib-cni-multus\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.517797 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517798 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-run-openvswitch\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.517868 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs podName:aa39c3a5-6b00-4d1d-b712-587e31db6d3d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:37.017832251 +0000 UTC m=+3.148245398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs") pod "network-metrics-daemon-w4chw" (UID: "aa39c3a5-6b00-4d1d-b712-587e31db6d3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.517882 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-systemd\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-systemd\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518191 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63087682-353b-454d-9cea-58c6e6a30613-etc-tuned\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.518240 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518217 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-os-release\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518258 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wnr\" (UniqueName: \"kubernetes.io/projected/63087682-353b-454d-9cea-58c6e6a30613-kube-api-access-47wnr\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518290 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7854f1f9-a33b-40b4-b4a5-cea927feb277-iptables-alerter-script\") pod \"iptables-alerter-sqsvb\" (UID: \"7854f1f9-a33b-40b4-b4a5-cea927feb277\") " pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-cnibin\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518313 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-os-release\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518324 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-systemd-units\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518383 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-cnibin\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518393 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518424 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-device-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518433 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-systemd-units\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-run-systemd\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518485 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-sysctl-d\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518501 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-run-systemd\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-multus-socket-dir-parent\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518567 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-var-lib-cni-bin\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.518768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518592 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b719a40-3782-4d09-9dd2-637375e15be2-multus-daemon-config\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518615 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-multus-socket-dir-parent\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518634 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-sysctl-d\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518635 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-etc-kubernetes\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518666 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-etc-kubernetes\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518676 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-var-lib-cni-bin\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-kubelet\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzk8\" (UniqueName: \"kubernetes.io/projected/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-kube-api-access-xrzk8\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518837 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-kubelet\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.518906 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-ovnkube-config\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519086 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-sysctl-conf\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd55f852-d386-432b-a490-583cc2a4db3d-hosts-file\") pod \"node-resolver-psmv2\" (UID: \"cd55f852-d386-432b-a490-583cc2a4db3d\") " pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519166 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbsn\" (UniqueName: \"kubernetes.io/projected/cd55f852-d386-432b-a490-583cc2a4db3d-kube-api-access-6mbsn\") pod \"node-resolver-psmv2\" (UID: \"cd55f852-d386-432b-a490-583cc2a4db3d\") " pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519184 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b719a40-3782-4d09-9dd2-637375e15be2-multus-daemon-config\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-cni-bin\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.519454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519240 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35748986-3b58-45b7-a1e0-a77c803c1ad3-kubelet-config\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519281 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519323 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519343 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-sysctl-conf\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-run\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519394 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-sys\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519401 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd55f852-d386-432b-a490-583cc2a4db3d-hosts-file\") pod \"node-resolver-psmv2\" (UID: \"cd55f852-d386-432b-a490-583cc2a4db3d\") " pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519418 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-env-overrides\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519447 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-ovnkube-config\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7854f1f9-a33b-40b4-b4a5-cea927feb277-host-slash\") pod \"iptables-alerter-sqsvb\" (UID: \"7854f1f9-a33b-40b4-b4a5-cea927feb277\") " pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519494 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-os-release\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519498 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7854f1f9-a33b-40b4-b4a5-cea927feb277-host-slash\") pod \"iptables-alerter-sqsvb\" (UID: \"7854f1f9-a33b-40b4-b4a5-cea927feb277\") " pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519525 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-registration-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519253 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-cni-bin\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520163 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519596 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-sys\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519602 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-run\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-modprobe-d\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519728 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63087682-353b-454d-9cea-58c6e6a30613-tmp\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-var-lib-openvswitch\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-node-log\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519819 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519846 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-etc-modprobe-d\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519883 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-ovn-node-metrics-cert\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519901 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519912 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-var-lib-openvswitch\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519937 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-node-log\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519971 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-socket-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.519988 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-env-overrides\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520014 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08452baf-e640-4628-8782-25b1cdb96215-agent-certs\") pod \"konnectivity-agent-s86zr\" (UID: \"08452baf-e640-4628-8782-25b1cdb96215\") " pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:36.520818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-lib-modules\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520081 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-run-k8s-cni-cncf-io\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520098 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63087682-353b-454d-9cea-58c6e6a30613-lib-modules\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520125 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-run-k8s-cni-cncf-io\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520135 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-etc-openvswitch\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520158 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/185cd267-f8e8-4e94-a620-9b5ac374fe19-serviceca\") pod \"node-ca-vd7wn\" (UID: \"185cd267-f8e8-4e94-a620-9b5ac374fe19\") " pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520173 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjvb\" (UniqueName: \"kubernetes.io/projected/8e61ffee-9a3a-46db-b61b-54804c0064e2-kube-api-access-nbjvb\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520194 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-run-netns\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520245 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dw24\" (UniqueName: \"kubernetes.io/projected/9b719a40-3782-4d09-9dd2-637375e15be2-kube-api-access-5dw24\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520263 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b719a40-3782-4d09-9dd2-637375e15be2-host-run-netns\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520288 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-etc-openvswitch\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520458 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63087682-353b-454d-9cea-58c6e6a30613-etc-tuned\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.521494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.520632 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/185cd267-f8e8-4e94-a620-9b5ac374fe19-serviceca\") pod \"node-ca-vd7wn\" (UID: \"185cd267-f8e8-4e94-a620-9b5ac374fe19\") " pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.525322 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.525289 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:36.525322 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.525312 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:36.525468 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.525327 2583 projected.go:196] Error preparing data for projected volume kube-api-access-zfzsg for pod openshift-network-diagnostics/network-check-target-xkxc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:36.525468 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.525387 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg podName:4f5b73dd-893c-4b4d-8b61-512994aaaa9d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:37.025370543 +0000 UTC m=+3.155783668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zfzsg" (UniqueName: "kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg") pod "network-check-target-xkxc8" (UID: "4f5b73dd-893c-4b4d-8b61-512994aaaa9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:36.527758 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.527697 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbfnh\" (UniqueName: \"kubernetes.io/projected/3b0053d1-a54c-4d0a-8b90-e37baf41c62f-kube-api-access-jbfnh\") pod \"ovnkube-node-wzqpt\" (UID: \"3b0053d1-a54c-4d0a-8b90-e37baf41c62f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.528158 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.527534 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgpx\" (UniqueName: \"kubernetes.io/projected/7854f1f9-a33b-40b4-b4a5-cea927feb277-kube-api-access-9pgpx\") pod \"iptables-alerter-sqsvb\" (UID: \"7854f1f9-a33b-40b4-b4a5-cea927feb277\") " pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.528789 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.528766 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57v6\" (UniqueName: \"kubernetes.io/projected/185cd267-f8e8-4e94-a620-9b5ac374fe19-kube-api-access-w57v6\") pod \"node-ca-vd7wn\" (UID: \"185cd267-f8e8-4e94-a620-9b5ac374fe19\") " pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.530642 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.530488 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wnr\" (UniqueName: \"kubernetes.io/projected/63087682-353b-454d-9cea-58c6e6a30613-kube-api-access-47wnr\") pod \"tuned-tthl2\" (UID: \"63087682-353b-454d-9cea-58c6e6a30613\") " pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.531534 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.531511 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dw24\" (UniqueName: \"kubernetes.io/projected/9b719a40-3782-4d09-9dd2-637375e15be2-kube-api-access-5dw24\") pod \"multus-tbs66\" (UID: \"9b719a40-3782-4d09-9dd2-637375e15be2\") " pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.531658 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.531636 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbsn\" (UniqueName: \"kubernetes.io/projected/cd55f852-d386-432b-a490-583cc2a4db3d-kube-api-access-6mbsn\") pod \"node-resolver-psmv2\" (UID: \"cd55f852-d386-432b-a490-583cc2a4db3d\") " pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.532461 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.532440 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzk8\" (UniqueName: \"kubernetes.io/projected/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-kube-api-access-xrzk8\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:36.551522 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.551501 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:36.621517 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621486 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:36.621676 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621632 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.621732 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.621678 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:36.621788 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621735 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.621788 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:36.621744 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret podName:35748986-3b58-45b7-a1e0-a77c803c1ad3 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:37.121726772 +0000 UTC m=+3.252139917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret") pod "global-pull-secret-syncer-mpk6m" (UID: "35748986-3b58-45b7-a1e0-a77c803c1ad3") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:36.621788 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621678 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mh5\" (UniqueName: \"kubernetes.io/projected/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-kube-api-access-t8mh5\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.621971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621796 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.621971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-sys-fs\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.621971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621857 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-cnibin\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.621971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621881 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.621971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.621971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621942 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-sys-fs\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.621971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621949 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-device-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.622311 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621986 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35748986-3b58-45b7-a1e0-a77c803c1ad3-kubelet-config\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:36.622311 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.621993 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.622311 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.622311 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622021 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.622311 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622036 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-cnibin\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.622311 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622050 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-os-release\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.622311 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-registration-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.622311 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622082 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35748986-3b58-45b7-a1e0-a77c803c1ad3-kubelet-config\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:36.622311 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622164 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-socket-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.622770 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622415 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.622770 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622476 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-device-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.622770 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622085 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-socket-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.622770 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622518 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjvb\" (UniqueName: \"kubernetes.io/projected/8e61ffee-9a3a-46db-b61b-54804c0064e2-kube-api-access-nbjvb\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.622770 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622565 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35748986-3b58-45b7-a1e0-a77c803c1ad3-dbus\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:36.622770 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622594 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-system-cni-dir\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.622770 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622701 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-system-cni-dir\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.622770 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622720 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.622770 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622715 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.623165 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.622848 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35748986-3b58-45b7-a1e0-a77c803c1ad3-dbus\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:36.623165 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.623037 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-os-release\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.623165 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.623142 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.623165 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.623144 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e61ffee-9a3a-46db-b61b-54804c0064e2-registration-dir\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.632973 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.632918 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8mh5\" (UniqueName: \"kubernetes.io/projected/aaea25d7-67d4-4e78-9966-2c76cb1ed2a3-kube-api-access-t8mh5\") pod \"multus-additional-cni-plugins-z2fx4\" (UID: \"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3\") " pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.633635 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.633611 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjvb\" (UniqueName: \"kubernetes.io/projected/8e61ffee-9a3a-46db-b61b-54804c0064e2-kube-api-access-nbjvb\") pod \"aws-ebs-csi-driver-node-dzdvg\" (UID: \"8e61ffee-9a3a-46db-b61b-54804c0064e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.698424 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.698162 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:41:36.705189 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.705158 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sqsvb" Jul 07 15:41:36.716927 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.716903 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tthl2" Jul 07 15:41:36.718036 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:36.718007 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7854f1f9_a33b_40b4_b4a5_cea927feb277.slice/crio-5151598eeb5697dbb4c363420be127a7acf58c0a8fab9dbc71766e53f5e7ec0f WatchSource:0}: Error finding container 5151598eeb5697dbb4c363420be127a7acf58c0a8fab9dbc71766e53f5e7ec0f: Status 404 returned error can't find the container with id 5151598eeb5697dbb4c363420be127a7acf58c0a8fab9dbc71766e53f5e7ec0f Jul 07 15:41:36.722142 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.721942 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vd7wn" Jul 07 15:41:36.727568 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.727396 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tbs66" Jul 07 15:41:36.728274 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:36.728240 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63087682_353b_454d_9cea_58c6e6a30613.slice/crio-21c1d17523d86f14ca7fbd1f6ffbbe506e4406b9e0f45b9a7b2ea5716278a1d2 WatchSource:0}: Error finding container 21c1d17523d86f14ca7fbd1f6ffbbe506e4406b9e0f45b9a7b2ea5716278a1d2: Status 404 returned error can't find the container with id 21c1d17523d86f14ca7fbd1f6ffbbe506e4406b9e0f45b9a7b2ea5716278a1d2 Jul 07 15:41:36.734250 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.732753 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-psmv2" Jul 07 15:41:36.740200 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.740175 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:36.743184 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:36.741835 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b719a40_3782_4d09_9dd2_637375e15be2.slice/crio-b8a89e0566ed2e64006b522fd794d3d772d35c980e631eaecb7be82fe19f32f0 WatchSource:0}: Error finding container b8a89e0566ed2e64006b522fd794d3d772d35c980e631eaecb7be82fe19f32f0: Status 404 returned error can't find the container with id b8a89e0566ed2e64006b522fd794d3d772d35c980e631eaecb7be82fe19f32f0 Jul 07 15:41:36.745922 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.745905 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" Jul 07 15:41:36.751721 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.751557 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" Jul 07 15:41:36.755948 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:36.755923 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08452baf_e640_4628_8782_25b1cdb96215.slice/crio-6aaba8055c5ffd41847b5515caec5e768f9b8d3184e6fb3b46f4d8437de42330 WatchSource:0}: Error finding container 6aaba8055c5ffd41847b5515caec5e768f9b8d3184e6fb3b46f4d8437de42330: Status 404 returned error can't find the container with id 6aaba8055c5ffd41847b5515caec5e768f9b8d3184e6fb3b46f4d8437de42330 Jul 07 15:41:36.764464 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:41:36.764441 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaea25d7_67d4_4e78_9966_2c76cb1ed2a3.slice/crio-f6af3bb0cef02628c31ed601802765444ac7771f81e8377f59f14b7a978e4ba3 WatchSource:0}: Error finding container f6af3bb0cef02628c31ed601802765444ac7771f81e8377f59f14b7a978e4ba3: Status 404 returned error can't find the container with id f6af3bb0cef02628c31ed601802765444ac7771f81e8377f59f14b7a978e4ba3 Jul 07 15:41:36.797281 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.797183 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s86zr" event={"ID":"08452baf-e640-4628-8782-25b1cdb96215","Type":"ContainerStarted","Data":"6aaba8055c5ffd41847b5515caec5e768f9b8d3184e6fb3b46f4d8437de42330"} Jul 07 15:41:36.798563 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.798522 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-psmv2" event={"ID":"cd55f852-d386-432b-a490-583cc2a4db3d","Type":"ContainerStarted","Data":"a2c07ca7c3be81e5aef06f2170428310e8ea8f6f7a6527c0d5911e864dda703d"} Jul 07 15:41:36.799960 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.799906 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tbs66" event={"ID":"9b719a40-3782-4d09-9dd2-637375e15be2","Type":"ContainerStarted","Data":"b8a89e0566ed2e64006b522fd794d3d772d35c980e631eaecb7be82fe19f32f0"} Jul 07 15:41:36.801160 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.801137 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tthl2" event={"ID":"63087682-353b-454d-9cea-58c6e6a30613","Type":"ContainerStarted","Data":"21c1d17523d86f14ca7fbd1f6ffbbe506e4406b9e0f45b9a7b2ea5716278a1d2"} Jul 07 15:41:36.802374 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.802338 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" event={"ID":"dbf149aa0654ab8e90cfec7925c91095","Type":"ContainerStarted","Data":"7ec035e8c0e527647bf0f10688ea8e7c47b53fd2301ca496038bd28a8ac81aaf"} Jul 07 15:41:36.803819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.803770 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" event={"ID":"3476c52a265caaabfc184395d47554a7","Type":"ContainerStarted","Data":"faa49dc7d41492e1fa7c72aaf79f74539f58f8d6855d60bbf7d99071e7cc0aa7"} Jul 07 15:41:36.805064 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.805006 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" event={"ID":"8e61ffee-9a3a-46db-b61b-54804c0064e2","Type":"ContainerStarted","Data":"536afea1b9259c23a47c2a24e43e4188d3e09e60a6d8acf825f4d99b259922b3"} Jul 07 15:41:36.806226 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.806204 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" event={"ID":"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3","Type":"ContainerStarted","Data":"f6af3bb0cef02628c31ed601802765444ac7771f81e8377f59f14b7a978e4ba3"} Jul 07 15:41:36.807291 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.807265 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vd7wn" event={"ID":"185cd267-f8e8-4e94-a620-9b5ac374fe19","Type":"ContainerStarted","Data":"5757079d1a56ea60626865f4edfc6430a808dcbfc17dd217f696f1eecb778497"} Jul 07 15:41:36.808314 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.808294 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sqsvb" event={"ID":"7854f1f9-a33b-40b4-b4a5-cea927feb277","Type":"ContainerStarted","Data":"5151598eeb5697dbb4c363420be127a7acf58c0a8fab9dbc71766e53f5e7ec0f"} Jul 07 15:41:36.809503 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:36.809485 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerStarted","Data":"ae53246af04be5c88060e290e10b5f6c416a4d03f7a82772b25c32c8b7205dcf"} Jul 07 15:41:37.024875 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:37.024791 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:37.025035 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.024949 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:37.025035 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.025019 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs podName:aa39c3a5-6b00-4d1d-b712-587e31db6d3d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:38.025001239 +0000 UTC m=+4.155414378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs") pod "network-metrics-daemon-w4chw" (UID: "aa39c3a5-6b00-4d1d-b712-587e31db6d3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:37.125587 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:37.125540 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:37.125755 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:37.125599 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:37.125755 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.125696 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:37.125755 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.125740 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:37.125755 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.125757 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:37.125937 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.125762 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret podName:35748986-3b58-45b7-a1e0-a77c803c1ad3 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:38.125743647 +0000 UTC m=+4.256156781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret") pod "global-pull-secret-syncer-mpk6m" (UID: "35748986-3b58-45b7-a1e0-a77c803c1ad3") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:37.125937 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.125770 2583 projected.go:196] Error preparing data for projected volume kube-api-access-zfzsg for pod openshift-network-diagnostics/network-check-target-xkxc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:37.125937 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.125802 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg podName:4f5b73dd-893c-4b4d-8b61-512994aaaa9d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:38.125793408 +0000 UTC m=+4.256206534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfzsg" (UniqueName: "kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg") pod "network-check-target-xkxc8" (UID: "4f5b73dd-893c-4b4d-8b61-512994aaaa9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:37.516614 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:37.516449 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:36:36 +0000 UTC" deadline="2028-02-28 01:47:01.015016985 +0000 UTC" Jul 07 15:41:37.516614 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:37.516506 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14410h5m23.498515152s" Jul 07 15:41:37.792830 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:37.792756 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:37.792988 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.792901 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:37.793386 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:37.793364 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:37.794169 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.793580 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:37.794169 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:37.793654 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:37.794169 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:37.793754 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:38.033485 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:38.032961 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:38.033485 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:38.033100 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:38.033485 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:38.033155 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs podName:aa39c3a5-6b00-4d1d-b712-587e31db6d3d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:40.033138061 +0000 UTC m=+6.163551190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs") pod "network-metrics-daemon-w4chw" (UID: "aa39c3a5-6b00-4d1d-b712-587e31db6d3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:38.134645 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:38.133801 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:38.134645 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:38.133845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:38.134645 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:38.134022 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:38.134645 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:38.134040 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:38.134645 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:38.134053 2583 projected.go:196] Error preparing data for projected volume kube-api-access-zfzsg for pod openshift-network-diagnostics/network-check-target-xkxc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:38.134645 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:38.134104 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg podName:4f5b73dd-893c-4b4d-8b61-512994aaaa9d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:40.134087455 +0000 UTC m=+6.264500584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfzsg" (UniqueName: "kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg") pod "network-check-target-xkxc8" (UID: "4f5b73dd-893c-4b4d-8b61-512994aaaa9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:38.134645 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:38.134495 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:38.134645 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:38.134565 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret podName:35748986-3b58-45b7-a1e0-a77c803c1ad3 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:40.134530803 +0000 UTC m=+6.264943929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret") pod "global-pull-secret-syncer-mpk6m" (UID: "35748986-3b58-45b7-a1e0-a77c803c1ad3") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:38.517455 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:38.517365 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:36:36 +0000 UTC" deadline="2027-12-08 20:28:16.884129496 +0000 UTC" Jul 07 15:41:38.517455 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:38.517405 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12460h46m38.366728837s" Jul 07 15:41:38.826667 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:38.826340 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" event={"ID":"dbf149aa0654ab8e90cfec7925c91095","Type":"ContainerStarted","Data":"479bd5a87a8a64bc487bd531e8a7bd1faac125c9e2e148a6aa87263560851702"} Jul 07 15:41:39.792712 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:39.792633 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:39.793139 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:39.792753 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:39.793202 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:39.793153 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:39.793433 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:39.793282 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:39.793433 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:39.793327 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:39.793618 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:39.793450 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:39.834180 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:39.834076 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" event={"ID":"3476c52a265caaabfc184395d47554a7","Type":"ContainerStarted","Data":"8dfcdd61903275d8fbfe5ffeebefcdd8fd91725d74edd6781123b6cbe721ca01"} Jul 07 15:41:39.854147 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:39.853815 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" podStartSLOduration=3.853798375 podStartE2EDuration="3.853798375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:38.845706771 +0000 UTC m=+4.976119919" watchObservedRunningTime="2026-07-07 15:41:39.853798375 +0000 UTC m=+5.984211524" Jul 07 15:41:40.046686 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:40.046593 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:40.046857 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:40.046724 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:40.046857 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:40.046786 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs podName:aa39c3a5-6b00-4d1d-b712-587e31db6d3d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:44.046766469 +0000 UTC m=+10.177179597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs") pod "network-metrics-daemon-w4chw" (UID: "aa39c3a5-6b00-4d1d-b712-587e31db6d3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:40.147614 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:40.147577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:40.147794 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:40.147628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:40.147854 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:40.147812 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:40.147854 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:40.147830 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:40.147854 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:40.147842 2583 projected.go:196] Error preparing data for projected volume kube-api-access-zfzsg for pod openshift-network-diagnostics/network-check-target-xkxc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:40.148008 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:40.147897 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg podName:4f5b73dd-893c-4b4d-8b61-512994aaaa9d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:44.147877488 +0000 UTC m=+10.278290612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfzsg" (UniqueName: "kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg") pod "network-check-target-xkxc8" (UID: "4f5b73dd-893c-4b4d-8b61-512994aaaa9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:40.148313 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:40.148276 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:40.148379 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:40.148325 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret podName:35748986-3b58-45b7-a1e0-a77c803c1ad3 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:44.148310444 +0000 UTC m=+10.278723573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret") pod "global-pull-secret-syncer-mpk6m" (UID: "35748986-3b58-45b7-a1e0-a77c803c1ad3") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:40.838261 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:40.838101 2583 generic.go:356] "Generic (PLEG): container finished" podID="3476c52a265caaabfc184395d47554a7" containerID="8dfcdd61903275d8fbfe5ffeebefcdd8fd91725d74edd6781123b6cbe721ca01" exitCode=0 Jul 07 15:41:40.838261 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:40.838157 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" event={"ID":"3476c52a265caaabfc184395d47554a7","Type":"ContainerDied","Data":"8dfcdd61903275d8fbfe5ffeebefcdd8fd91725d74edd6781123b6cbe721ca01"} Jul 07 15:41:41.792456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:41.792420 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:41.792622 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:41.792578 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:41.792622 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:41.792586 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:41.792703 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:41.792675 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:41.792975 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:41.792957 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:41.793093 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:41.793075 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:43.792581 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:43.792441 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:43.793038 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:43.792598 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:43.793038 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:43.792886 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:43.793038 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:43.792959 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:43.793200 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:43.793049 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:43.793200 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:43.793109 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:44.081010 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:44.080924 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:44.081176 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:44.081099 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:44.081241 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:44.081185 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs podName:aa39c3a5-6b00-4d1d-b712-587e31db6d3d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:52.081164501 +0000 UTC m=+18.211577626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs") pod "network-metrics-daemon-w4chw" (UID: "aa39c3a5-6b00-4d1d-b712-587e31db6d3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:44.181823 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:44.181789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:44.181823 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:44.181829 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:44.182050 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:44.181967 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:44.182050 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:44.181978 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:44.182050 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:44.181992 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:44.182050 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:44.182003 2583 projected.go:196] Error preparing data for projected volume kube-api-access-zfzsg for pod openshift-network-diagnostics/network-check-target-xkxc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:44.182050 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:44.182033 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret podName:35748986-3b58-45b7-a1e0-a77c803c1ad3 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:52.182014992 +0000 UTC m=+18.312428136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret") pod "global-pull-secret-syncer-mpk6m" (UID: "35748986-3b58-45b7-a1e0-a77c803c1ad3") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:44.182050 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:44.182051 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg podName:4f5b73dd-893c-4b4d-8b61-512994aaaa9d nodeName:}" failed. No retries permitted until 2026-07-07 15:41:52.182041998 +0000 UTC m=+18.312455122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfzsg" (UniqueName: "kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg") pod "network-check-target-xkxc8" (UID: "4f5b73dd-893c-4b4d-8b61-512994aaaa9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:45.792504 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:45.792466 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:45.792978 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:45.792630 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:45.793054 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:45.793031 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:45.793289 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:45.793134 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:45.793289 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:45.793182 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:45.793289 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:45.793243 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:47.792108 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:47.792072 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:47.792108 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:47.792109 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:47.792574 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:47.792119 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:47.792574 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:47.792200 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:47.792574 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:47.792357 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:47.792574 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:47.792448 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:49.792749 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:49.792682 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:49.793305 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:49.792795 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:49.793305 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:49.792809 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:49.793305 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:49.792823 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:49.793305 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:49.792886 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:49.793305 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:49.792966 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:51.792179 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:51.792101 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:51.792179 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:51.792138 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:51.792780 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:51.792207 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:51.792780 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:51.792222 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:51.792780 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:51.792287 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:51.792780 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:51.792351 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:52.140656 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:52.140586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:52.140796 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:52.140728 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:52.140796 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:52.140789 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs podName:aa39c3a5-6b00-4d1d-b712-587e31db6d3d nodeName:}" failed. No retries permitted until 2026-07-07 15:42:08.140773455 +0000 UTC m=+34.271186583 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs") pod "network-metrics-daemon-w4chw" (UID: "aa39c3a5-6b00-4d1d-b712-587e31db6d3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:52.241347 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:52.241304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:52.241515 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:52.241360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:52.241515 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:52.241425 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:52.241515 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:52.241504 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret podName:35748986-3b58-45b7-a1e0-a77c803c1ad3 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:08.24148699 +0000 UTC m=+34.371900127 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret") pod "global-pull-secret-syncer-mpk6m" (UID: "35748986-3b58-45b7-a1e0-a77c803c1ad3") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:52.241705 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:52.241526 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:52.241705 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:52.241557 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:52.241705 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:52.241571 2583 projected.go:196] Error preparing data for projected volume kube-api-access-zfzsg for pod openshift-network-diagnostics/network-check-target-xkxc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:52.241705 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:52.241618 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg podName:4f5b73dd-893c-4b4d-8b61-512994aaaa9d nodeName:}" failed. No retries permitted until 2026-07-07 15:42:08.241604635 +0000 UTC m=+34.372017761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfzsg" (UniqueName: "kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg") pod "network-check-target-xkxc8" (UID: "4f5b73dd-893c-4b4d-8b61-512994aaaa9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:53.792506 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:53.792466 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:53.792506 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:53.792506 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:53.792985 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:53.792589 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:53.792985 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:53.792619 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:53.792985 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:53.792679 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:53.792985 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:53.792755 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:55.792635 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:55.792604 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:55.792635 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:55.792622 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:55.793141 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:55.792622 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:55.793141 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:55.792722 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:55.793141 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:55.792767 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:55.793141 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:55.792869 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:56.870496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.870286 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" event={"ID":"3476c52a265caaabfc184395d47554a7","Type":"ContainerStarted","Data":"ba20fccc609adc745cc8239787e3d8b2592eb1de4c0e5ae6f2440d2d92f56e83"} Jul 07 15:41:56.871879 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.871855 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" event={"ID":"8e61ffee-9a3a-46db-b61b-54804c0064e2","Type":"ContainerStarted","Data":"1e424a2f30dc17be6c65b5a7d53f4ce861344518ae4992c68d31febbad902b48"} Jul 07 15:41:56.873312 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.873288 2583 generic.go:356] "Generic (PLEG): container finished" podID="aaea25d7-67d4-4e78-9966-2c76cb1ed2a3" containerID="756aeabf0f00dfc8982eed7572ad7bc1c9f685e87c962d27739114dfb31db143" exitCode=0 Jul 07 15:41:56.873422 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.873353 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" event={"ID":"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3","Type":"ContainerDied","Data":"756aeabf0f00dfc8982eed7572ad7bc1c9f685e87c962d27739114dfb31db143"} Jul 07 15:41:56.875438 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.874966 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vd7wn" event={"ID":"185cd267-f8e8-4e94-a620-9b5ac374fe19","Type":"ContainerStarted","Data":"19e1b5da406b33b153e80ef3a0b7709fa3d44276e1480a24fc5441517c382e93"} Jul 07 15:41:56.877139 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.877122 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:41:56.877497 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.877477 2583 generic.go:356] "Generic (PLEG): container finished" podID="3b0053d1-a54c-4d0a-8b90-e37baf41c62f" containerID="787d7b8b243b0fd498954bad27d27e44ab7bb065bd258eafbe0866ad87265f95" exitCode=1 Jul 07 15:41:56.877615 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.877560 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerDied","Data":"787d7b8b243b0fd498954bad27d27e44ab7bb065bd258eafbe0866ad87265f95"} Jul 07 15:41:56.877675 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.877629 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerStarted","Data":"088e97c0c371007535293b42f67ffa4d4765373d2322e7333325ba16a285619e"} Jul 07 15:41:56.879112 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.879076 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s86zr" event={"ID":"08452baf-e640-4628-8782-25b1cdb96215","Type":"ContainerStarted","Data":"4579b087d564d445b4b6c472e95b5009a28a2c39eade5c9956f24ddd6d36ab12"} Jul 07 15:41:56.881510 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.881484 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-psmv2" event={"ID":"cd55f852-d386-432b-a490-583cc2a4db3d","Type":"ContainerStarted","Data":"4ccc7fad0fed0c0a874790e2d494cf06c6ff1f1b88779c2c9c4a4ecbda41f1cb"} Jul 07 15:41:56.885566 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.885370 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tbs66" event={"ID":"9b719a40-3782-4d09-9dd2-637375e15be2","Type":"ContainerStarted","Data":"eba2a80c23a974a6482e317c45e71c62addb48ab1037e6593080ecfdce582af0"} Jul 07 15:41:56.887060 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.887034 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tthl2" event={"ID":"63087682-353b-454d-9cea-58c6e6a30613","Type":"ContainerStarted","Data":"8fd6db210158a1e0b0022f80419ba1b06adbd0855b3a0ebe8cf2029d1f73047b"} Jul 07 15:41:56.891801 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.889377 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" podStartSLOduration=20.889361851 podStartE2EDuration="20.889361851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:56.888904393 +0000 UTC m=+23.019317542" watchObservedRunningTime="2026-07-07 15:41:56.889361851 +0000 UTC m=+23.019774998" Jul 07 15:41:56.907600 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.907540 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tthl2" podStartSLOduration=3.719164751 podStartE2EDuration="22.9075266s" totalImagesPullingTime="19.188361849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:56.907351587 +0000 UTC m=+23.037764734" watchObservedRunningTime="2026-07-07 15:41:56.9075266 +0000 UTC m=+23.037939746" Jul 07 15:41:56.923839 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.923789 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vd7wn" podStartSLOduration=12.05405898 podStartE2EDuration="22.923775006s" totalImagesPullingTime="10.869716026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:56.923244156 +0000 UTC m=+23.053657313" watchObservedRunningTime="2026-07-07 15:41:56.923775006 +0000 UTC m=+23.054188152" Jul 07 15:41:56.938978 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.938934 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-psmv2" podStartSLOduration=3.768436588 podStartE2EDuration="22.938922065s" totalImagesPullingTime="19.170485477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:56.938362895 +0000 UTC m=+23.068776041" watchObservedRunningTime="2026-07-07 15:41:56.938922065 +0000 UTC m=+23.069335211" Jul 07 15:41:56.979623 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.979581 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tbs66" podStartSLOduration=3.720929785 podStartE2EDuration="22.979566422s" totalImagesPullingTime="19.258636637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:56.979360198 +0000 UTC m=+23.109773344" watchObservedRunningTime="2026-07-07 15:41:56.979566422 +0000 UTC m=+23.109979570" Jul 07 15:41:56.996523 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:56.996485 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s86zr" podStartSLOduration=2.838327631 podStartE2EDuration="21.996472551s" totalImagesPullingTime="19.15814492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:56.996104776 +0000 UTC m=+23.126517923" watchObservedRunningTime="2026-07-07 15:41:56.996472551 +0000 UTC m=+23.126885697" Jul 07 15:41:57.297908 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.297809 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 15:41:57.552670 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.552603 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:57.553191 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.553175 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:57.668801 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.668681 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T15:41:57.297827605Z","UUID":"b6e8a213-6b06-4215-9203-d56bfb65c1a8","Handler":null,"Name":"","Endpoint":""} Jul 07 15:41:57.671650 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.671630 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 15:41:57.671779 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.671659 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 15:41:57.792182 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.792150 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:57.792418 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.792150 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:57.792418 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:57.792268 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:57.792418 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.792153 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:57.792418 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:57.792344 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:57.792660 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:57.792416 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:57.891244 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.891167 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" event={"ID":"8e61ffee-9a3a-46db-b61b-54804c0064e2","Type":"ContainerStarted","Data":"ad39aa852c55e8c5ccbd0f42d455ee49b3786f8ca09377ad7d7e2416bff9b777"} Jul 07 15:41:57.892455 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.892431 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sqsvb" event={"ID":"7854f1f9-a33b-40b4-b4a5-cea927feb277","Type":"ContainerStarted","Data":"d0c14bff6c7830395cc50026985c7e56ed4c2d1e6825f6f1c7625e16bddcbdc0"} Jul 07 15:41:57.897099 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.897080 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:41:57.897457 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.897428 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerStarted","Data":"9d35d1f6f74ec85fba8d89b198f036f15bef5d8b67936dcde2a29678becde1c1"} Jul 07 15:41:57.897539 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.897494 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerStarted","Data":"da9312244e680a27cc826719b982a0bee5f3a18835a4b4feeeafa65164b09c08"} Jul 07 15:41:57.897921 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.897901 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:57.898033 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.897937 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerStarted","Data":"c307113d235fb7c4bfc8ccb56a92dc20a9822dbea92d6e1de7d9c94e10d89dbf"} Jul 07 15:41:57.898033 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.897989 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerStarted","Data":"28889c301d184f6405917f3392df02c5f66b65fb1b406aa2a6df4c0fe3841642"} Jul 07 15:41:57.898131 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.898074 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s86zr" Jul 07 15:41:57.909723 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:57.909673 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sqsvb" podStartSLOduration=4.710833334 podStartE2EDuration="23.909657031s" totalImagesPullingTime="19.198823697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:57.909301367 +0000 UTC m=+24.039714524" watchObservedRunningTime="2026-07-07 15:41:57.909657031 +0000 UTC m=+24.040070178" Jul 07 15:41:59.792750 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:59.792521 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:41:59.792750 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:59.792586 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:41:59.793254 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:59.792586 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:41:59.793254 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:59.792807 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:41:59.793254 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:59.792896 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:41:59.793254 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:41:59.793010 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:41:59.903814 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:59.903786 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:41:59.904157 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:59.904134 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerStarted","Data":"12430fc745cfac3d5b59f8a2cb2635958077ec34106d47ca91cd52573c4cf63e"} Jul 07 15:41:59.906140 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:41:59.906114 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" event={"ID":"8e61ffee-9a3a-46db-b61b-54804c0064e2","Type":"ContainerStarted","Data":"0a8de47ad203c1d6ace66c16939d77bc6fe53e47cee41fe181027c6af3622126"} Jul 07 15:42:01.792941 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:01.792862 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:01.792941 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:01.792901 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:01.793472 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:01.792977 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:01.793472 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:01.792992 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:01.793472 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:01.793053 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:01.793472 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:01.793138 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:02.913626 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.913427 2583 generic.go:356] "Generic (PLEG): container finished" podID="aaea25d7-67d4-4e78-9966-2c76cb1ed2a3" containerID="91afa133325956ffd45fa4fbc5577b02a325d95bd61603958f5d021351107ed8" exitCode=0 Jul 07 15:42:02.914254 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.913504 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" event={"ID":"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3","Type":"ContainerDied","Data":"91afa133325956ffd45fa4fbc5577b02a325d95bd61603958f5d021351107ed8"} Jul 07 15:42:02.917082 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.917062 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:42:02.917420 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.917401 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerStarted","Data":"cfd6cfe57d3f9d0ce37a698718626b78aef381bc81c6fd1d586dcb6ebbbcd042"} Jul 07 15:42:02.917736 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.917715 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:42:02.917835 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.917773 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:42:02.917835 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.917822 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:42:02.917931 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.917892 2583 scope.go:117] "RemoveContainer" containerID="787d7b8b243b0fd498954bad27d27e44ab7bb065bd258eafbe0866ad87265f95" Jul 07 15:42:02.933733 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.933709 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:42:02.933837 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.933825 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:42:02.946910 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:02.946872 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzdvg" podStartSLOduration=6.05345969 podStartE2EDuration="27.94685707s" totalImagesPullingTime="21.89339738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:59.93022506 +0000 UTC m=+26.060638205" watchObservedRunningTime="2026-07-07 15:42:02.94685707 +0000 UTC m=+29.077270221" Jul 07 15:42:03.792073 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:03.792038 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:03.792317 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:03.792038 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:03.792317 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:03.792149 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:03.792317 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:03.792225 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:03.792317 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:03.792038 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:03.792317 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:03.792309 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:03.923593 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:03.923565 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:42:03.923983 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:03.923949 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" event={"ID":"3b0053d1-a54c-4d0a-8b90-e37baf41c62f","Type":"ContainerStarted","Data":"69c2b2feafefb42dc28deb061dbc77911206f93a1ed1accaafdb1762275c8a6b"} Jul 07 15:42:03.957582 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:03.957515 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" podStartSLOduration=10.045961222 podStartE2EDuration="29.957500327s" totalImagesPullingTime="19.911539105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:03.955937716 +0000 UTC m=+30.086350864" watchObservedRunningTime="2026-07-07 15:42:03.957500327 +0000 UTC m=+30.087913472" Jul 07 15:42:04.928181 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:04.928141 2583 generic.go:356] "Generic (PLEG): container finished" podID="aaea25d7-67d4-4e78-9966-2c76cb1ed2a3" containerID="d0a4dfabb154142cdcf207d9389e0f0ed5e2b3207ad4ca350685180d38f781bc" exitCode=0 Jul 07 15:42:04.928577 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:04.928225 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" event={"ID":"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3","Type":"ContainerDied","Data":"d0a4dfabb154142cdcf207d9389e0f0ed5e2b3207ad4ca350685180d38f781bc"} Jul 07 15:42:05.792576 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:05.792524 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:05.792760 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:05.792530 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:05.792760 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:05.792663 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:05.792909 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:05.792773 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:05.792909 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:05.792530 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:05.792909 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:05.792881 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:06.934480 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:06.934450 2583 generic.go:356] "Generic (PLEG): container finished" podID="aaea25d7-67d4-4e78-9966-2c76cb1ed2a3" containerID="bbe089910c1659b2533bb2a2ba35eedfa121a8e1b9cf9d21d50b2fc2c4291f95" exitCode=0 Jul 07 15:42:06.934883 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:06.934505 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" event={"ID":"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3","Type":"ContainerDied","Data":"bbe089910c1659b2533bb2a2ba35eedfa121a8e1b9cf9d21d50b2fc2c4291f95"} Jul 07 15:42:07.792542 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:07.792505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:07.792732 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:07.792505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:07.792732 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:07.792570 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:07.792732 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:07.792667 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:07.792875 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:07.792765 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:07.792875 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:07.792841 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:08.160489 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:08.160409 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:08.160904 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:08.160575 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:42:08.160904 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:08.160641 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs podName:aa39c3a5-6b00-4d1d-b712-587e31db6d3d nodeName:}" failed. No retries permitted until 2026-07-07 15:42:40.160625726 +0000 UTC m=+66.291038850 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs") pod "network-metrics-daemon-w4chw" (UID: "aa39c3a5-6b00-4d1d-b712-587e31db6d3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:42:08.261854 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:08.261819 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:08.261854 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:08.261854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:08.262046 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:08.261959 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:42:08.262046 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:08.261963 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:42:08.262046 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:08.262027 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret podName:35748986-3b58-45b7-a1e0-a77c803c1ad3 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:40.262012056 +0000 UTC m=+66.392425180 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret") pod "global-pull-secret-syncer-mpk6m" (UID: "35748986-3b58-45b7-a1e0-a77c803c1ad3") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:42:08.262176 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:08.261972 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:42:08.262176 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:08.262068 2583 projected.go:196] Error preparing data for projected volume kube-api-access-zfzsg for pod openshift-network-diagnostics/network-check-target-xkxc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:42:08.262176 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:08.262106 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg podName:4f5b73dd-893c-4b4d-8b61-512994aaaa9d nodeName:}" failed. No retries permitted until 2026-07-07 15:42:40.262096433 +0000 UTC m=+66.392509557 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfzsg" (UniqueName: "kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg") pod "network-check-target-xkxc8" (UID: "4f5b73dd-893c-4b4d-8b61-512994aaaa9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:42:09.792757 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:09.792492 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:09.793226 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:09.792492 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:09.793226 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:09.792860 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:09.793226 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:09.792492 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:09.793226 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:09.792957 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:09.793226 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:09.793002 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:11.792771 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:11.792724 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:11.793159 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:11.792727 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:11.793159 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:11.792833 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:11.793159 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:11.792919 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:11.793159 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:11.792727 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:11.793159 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:11.792978 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:11.948602 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:11.948573 2583 generic.go:356] "Generic (PLEG): container finished" podID="aaea25d7-67d4-4e78-9966-2c76cb1ed2a3" containerID="83e2f3e2e799eacadef56ee70d989bfb0e76ae5958e7c2a2b22173b52a737a82" exitCode=0 Jul 07 15:42:11.948737 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:11.948627 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" event={"ID":"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3","Type":"ContainerDied","Data":"83e2f3e2e799eacadef56ee70d989bfb0e76ae5958e7c2a2b22173b52a737a82"} Jul 07 15:42:12.953732 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:12.953699 2583 generic.go:356] "Generic (PLEG): container finished" podID="aaea25d7-67d4-4e78-9966-2c76cb1ed2a3" containerID="7051128dd67c3ca5e19dd35f769f4aaf59491b6084cceb4194ce23a177a96ec2" exitCode=0 Jul 07 15:42:12.954062 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:12.953758 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" event={"ID":"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3","Type":"ContainerDied","Data":"7051128dd67c3ca5e19dd35f769f4aaf59491b6084cceb4194ce23a177a96ec2"} Jul 07 15:42:13.791946 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:13.791914 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:13.791946 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:13.791944 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:13.792188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:13.791914 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:13.792188 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:13.792034 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:13.792188 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:13.792090 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:13.792188 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:13.792175 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:13.959433 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:13.959391 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" event={"ID":"aaea25d7-67d4-4e78-9966-2c76cb1ed2a3","Type":"ContainerStarted","Data":"50db0ba2b71811e65147bb616ac2d8f0903096be98b04ba9aefd2f3dd6af30f2"} Jul 07 15:42:13.987449 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:13.987400 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z2fx4" podStartSLOduration=7.92201459 podStartE2EDuration="38.987384337s" totalImagesPullingTime="31.065369747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:13.985951249 +0000 UTC m=+40.116364396" watchObservedRunningTime="2026-07-07 15:42:13.987384337 +0000 UTC m=+40.117797482" Jul 07 15:42:15.792117 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:15.792080 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:15.792588 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:15.792080 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:15.792588 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:15.792183 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:15.792588 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:15.792202 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:15.792588 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:15.792326 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:15.792588 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:15.792419 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:17.792865 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:17.792832 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:17.793328 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:17.792832 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:17.793328 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:17.792944 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:17.793328 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:17.793025 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:17.793328 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:17.792833 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:17.793328 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:17.793102 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:19.792877 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:19.792839 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:19.793408 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:19.792894 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:19.793408 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:19.792858 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:19.793408 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:19.792957 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:19.793408 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:19.793026 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:19.793408 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:19.793095 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:19.971619 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:19.971587 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mpk6m"] Jul 07 15:42:19.972605 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:19.972582 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:19.972712 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:19.972676 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:19.974392 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:19.974366 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xkxc8"] Jul 07 15:42:19.974539 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:19.974451 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:19.974629 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:19.974567 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:19.975020 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:19.975002 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w4chw"] Jul 07 15:42:19.975103 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:19.975073 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:19.975160 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:19.975149 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:21.791847 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:21.791816 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:21.792354 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:21.791856 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:21.792354 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:21.791816 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:21.792354 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:21.791926 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:21.792354 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:21.792037 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:21.792354 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:21.792098 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:23.792422 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:23.792236 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:23.792899 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:23.792290 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:23.792899 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:23.792504 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkxc8" podUID="4f5b73dd-893c-4b4d-8b61-512994aaaa9d" Jul 07 15:42:23.792899 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:23.792582 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpk6m" podUID="35748986-3b58-45b7-a1e0-a77c803c1ad3" Jul 07 15:42:23.792899 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:23.792299 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:23.792899 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:23.792661 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4chw" podUID="aa39c3a5-6b00-4d1d-b712-587e31db6d3d" Jul 07 15:42:24.935867 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:24.935787 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeReady" Jul 07 15:42:24.936209 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:24.935939 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 15:42:25.001066 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.001026 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mgwz7"] Jul 07 15:42:25.015641 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.015604 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r7ldp"] Jul 07 15:42:25.015796 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.015741 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.019261 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.019201 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 15:42:25.019261 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.019241 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 15:42:25.019501 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.019484 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w75vl\"" Jul 07 15:42:25.029429 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.029411 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgwz7"] Jul 07 15:42:25.029506 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.029450 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r7ldp"] Jul 07 15:42:25.029569 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.029516 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.032262 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.032237 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 15:42:25.032368 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.032336 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 15:42:25.032368 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.032335 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 15:42:25.032464 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.032382 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4xbb9\"" Jul 07 15:42:25.032464 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.032382 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 15:42:25.103794 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.103765 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-66js4"] Jul 07 15:42:25.125396 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.125370 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-66js4"] Jul 07 15:42:25.125541 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.125478 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-66js4" Jul 07 15:42:25.128381 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.128358 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 15:42:25.128513 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.128361 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 15:42:25.128513 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.128398 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q5k6l\"" Jul 07 15:42:25.128513 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.128392 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 15:42:25.178440 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.178399 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5tdb\" (UniqueName: \"kubernetes.io/projected/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-kube-api-access-j5tdb\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.178647 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.178463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.178647 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.178483 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bc8fd0-fce0-499d-a241-2df1853d1f60-tmp-dir\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.178647 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.178500 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppxmt\" (UniqueName: \"kubernetes.io/projected/b0bc8fd0-fce0-499d-a241-2df1853d1f60-kube-api-access-ppxmt\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.178647 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.178606 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bc8fd0-fce0-499d-a241-2df1853d1f60-config-volume\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.178647 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.178641 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-crio-socket\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.178822 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.178672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-data-volume\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.178822 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.178690 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.178822 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.178715 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0bc8fd0-fce0-499d-a241-2df1853d1f60-metrics-tls\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.279795 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.279720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bc8fd0-fce0-499d-a241-2df1853d1f60-config-volume\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.279795 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.279757 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-crio-socket\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.279795 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.279779 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvm4\" (UniqueName: \"kubernetes.io/projected/3d1ee745-d61d-4d6d-bda8-6eb2417bd150-kube-api-access-8lvm4\") pod \"ingress-canary-66js4\" (UID: \"3d1ee745-d61d-4d6d-bda8-6eb2417bd150\") " pod="openshift-ingress-canary/ingress-canary-66js4" Jul 07 15:42:25.279984 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.279803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-data-volume\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.279984 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.279825 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.279984 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.279849 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0bc8fd0-fce0-499d-a241-2df1853d1f60-metrics-tls\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.279984 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.279878 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5tdb\" (UniqueName: \"kubernetes.io/projected/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-kube-api-access-j5tdb\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.280176 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.279977 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-crio-socket\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.280176 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.280069 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.280176 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.280113 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bc8fd0-fce0-499d-a241-2df1853d1f60-tmp-dir\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.280176 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.280142 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppxmt\" (UniqueName: \"kubernetes.io/projected/b0bc8fd0-fce0-499d-a241-2df1853d1f60-kube-api-access-ppxmt\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.280176 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.280169 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-data-volume\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.280399 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.280229 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d1ee745-d61d-4d6d-bda8-6eb2417bd150-cert\") pod \"ingress-canary-66js4\" (UID: \"3d1ee745-d61d-4d6d-bda8-6eb2417bd150\") " pod="openshift-ingress-canary/ingress-canary-66js4" Jul 07 15:42:25.280456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.280445 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bc8fd0-fce0-499d-a241-2df1853d1f60-tmp-dir\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.280496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.280482 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bc8fd0-fce0-499d-a241-2df1853d1f60-config-volume\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.280686 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.280666 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.283946 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.283928 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.284008 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.283951 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0bc8fd0-fce0-499d-a241-2df1853d1f60-metrics-tls\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.290078 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.290051 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppxmt\" (UniqueName: \"kubernetes.io/projected/b0bc8fd0-fce0-499d-a241-2df1853d1f60-kube-api-access-ppxmt\") pod \"dns-default-mgwz7\" (UID: \"b0bc8fd0-fce0-499d-a241-2df1853d1f60\") " pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.290255 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.290230 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5tdb\" (UniqueName: \"kubernetes.io/projected/7aae2321-d8a2-4a29-a053-84d7d79b1f6a-kube-api-access-j5tdb\") pod \"insights-runtime-extractor-r7ldp\" (UID: \"7aae2321-d8a2-4a29-a053-84d7d79b1f6a\") " pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.326342 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.326308 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:25.337486 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.337468 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r7ldp" Jul 07 15:42:25.381412 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.381385 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d1ee745-d61d-4d6d-bda8-6eb2417bd150-cert\") pod \"ingress-canary-66js4\" (UID: \"3d1ee745-d61d-4d6d-bda8-6eb2417bd150\") " pod="openshift-ingress-canary/ingress-canary-66js4" Jul 07 15:42:25.381532 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.381428 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvm4\" (UniqueName: \"kubernetes.io/projected/3d1ee745-d61d-4d6d-bda8-6eb2417bd150-kube-api-access-8lvm4\") pod \"ingress-canary-66js4\" (UID: \"3d1ee745-d61d-4d6d-bda8-6eb2417bd150\") " pod="openshift-ingress-canary/ingress-canary-66js4" Jul 07 15:42:25.383959 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.383931 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d1ee745-d61d-4d6d-bda8-6eb2417bd150-cert\") pod \"ingress-canary-66js4\" (UID: \"3d1ee745-d61d-4d6d-bda8-6eb2417bd150\") " pod="openshift-ingress-canary/ingress-canary-66js4" Jul 07 15:42:25.392916 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.392894 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvm4\" (UniqueName: \"kubernetes.io/projected/3d1ee745-d61d-4d6d-bda8-6eb2417bd150-kube-api-access-8lvm4\") pod \"ingress-canary-66js4\" (UID: \"3d1ee745-d61d-4d6d-bda8-6eb2417bd150\") " pod="openshift-ingress-canary/ingress-canary-66js4" Jul 07 15:42:25.433740 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.433706 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-66js4" Jul 07 15:42:25.532208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.532080 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r7ldp"] Jul 07 15:42:25.541005 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.540921 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgwz7"] Jul 07 15:42:25.542281 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:25.542232 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aae2321_d8a2_4a29_a053_84d7d79b1f6a.slice/crio-1f93b1d88e5911e94c676ed1c34c893714f66040e6ab7e0f2290a034b5c0c13a WatchSource:0}: Error finding container 1f93b1d88e5911e94c676ed1c34c893714f66040e6ab7e0f2290a034b5c0c13a: Status 404 returned error can't find the container with id 1f93b1d88e5911e94c676ed1c34c893714f66040e6ab7e0f2290a034b5c0c13a Jul 07 15:42:25.545413 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:25.545380 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bc8fd0_fce0_499d_a241_2df1853d1f60.slice/crio-5544f1017c891e255bf8960ae796ec55a61d93e38905f06f9ddc2a168c8244fb WatchSource:0}: Error finding container 5544f1017c891e255bf8960ae796ec55a61d93e38905f06f9ddc2a168c8244fb: Status 404 returned error can't find the container with id 5544f1017c891e255bf8960ae796ec55a61d93e38905f06f9ddc2a168c8244fb Jul 07 15:42:25.607441 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.607411 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-66js4"] Jul 07 15:42:25.610700 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:25.610679 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1ee745_d61d_4d6d_bda8_6eb2417bd150.slice/crio-f4cff673a9daec77ffcb3dbf885d821f675092f8a179007f49574ad63781bbef WatchSource:0}: Error finding container f4cff673a9daec77ffcb3dbf885d821f675092f8a179007f49574ad63781bbef: Status 404 returned error can't find the container with id f4cff673a9daec77ffcb3dbf885d821f675092f8a179007f49574ad63781bbef Jul 07 15:42:25.792815 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.792744 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:25.792815 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.792761 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:25.792815 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.792782 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:25.795988 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.795971 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 15:42:25.795988 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.795977 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t2q2r\"" Jul 07 15:42:25.796269 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.796252 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 15:42:25.796346 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.796288 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k55hz\"" Jul 07 15:42:25.796346 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.796289 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 15:42:25.796447 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.796371 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 15:42:25.985804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.985769 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-66js4" event={"ID":"3d1ee745-d61d-4d6d-bda8-6eb2417bd150","Type":"ContainerStarted","Data":"f4cff673a9daec77ffcb3dbf885d821f675092f8a179007f49574ad63781bbef"} Jul 07 15:42:25.986948 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.986924 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgwz7" event={"ID":"b0bc8fd0-fce0-499d-a241-2df1853d1f60","Type":"ContainerStarted","Data":"5544f1017c891e255bf8960ae796ec55a61d93e38905f06f9ddc2a168c8244fb"} Jul 07 15:42:25.987947 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:25.987928 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r7ldp" event={"ID":"7aae2321-d8a2-4a29-a053-84d7d79b1f6a","Type":"ContainerStarted","Data":"1f93b1d88e5911e94c676ed1c34c893714f66040e6ab7e0f2290a034b5c0c13a"} Jul 07 15:42:28.263897 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.263857 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br"] Jul 07 15:42:28.285621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.285594 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br"] Jul 07 15:42:28.285760 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.285715 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.291288 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.291265 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 15:42:28.291423 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.291404 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 15:42:28.291482 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.291410 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 15:42:28.292424 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.292208 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 15:42:28.292424 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.292247 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pzz44\"" Jul 07 15:42:28.292424 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.292298 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 15:42:28.406153 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.406112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spkn5\" (UniqueName: \"kubernetes.io/projected/bed3c459-1a4f-4517-9939-ac95d82c1dc1-kube-api-access-spkn5\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.406335 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.406204 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bed3c459-1a4f-4517-9939-ac95d82c1dc1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.406335 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.406252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed3c459-1a4f-4517-9939-ac95d82c1dc1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.406444 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.406354 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bed3c459-1a4f-4517-9939-ac95d82c1dc1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.507793 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.507747 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed3c459-1a4f-4517-9939-ac95d82c1dc1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.507971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.507844 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bed3c459-1a4f-4517-9939-ac95d82c1dc1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.507971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.507892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spkn5\" (UniqueName: \"kubernetes.io/projected/bed3c459-1a4f-4517-9939-ac95d82c1dc1-kube-api-access-spkn5\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.507971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.507953 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bed3c459-1a4f-4517-9939-ac95d82c1dc1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.508634 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.508541 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed3c459-1a4f-4517-9939-ac95d82c1dc1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.511849 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.511823 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bed3c459-1a4f-4517-9939-ac95d82c1dc1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.511996 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.511867 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bed3c459-1a4f-4517-9939-ac95d82c1dc1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.520573 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.520486 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spkn5\" (UniqueName: \"kubernetes.io/projected/bed3c459-1a4f-4517-9939-ac95d82c1dc1-kube-api-access-spkn5\") pod \"prometheus-operator-567f7b8b7c-bk4br\" (UID: \"bed3c459-1a4f-4517-9939-ac95d82c1dc1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:28.596463 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:28.596431 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" Jul 07 15:42:30.840799 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.840716 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br"] Jul 07 15:42:30.889388 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.889358 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c89d7c965-rtqd9"] Jul 07 15:42:30.896561 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.896529 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:30.900276 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.900257 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 15:42:30.900630 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.900612 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 15:42:30.900776 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.900765 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 15:42:30.900916 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.900897 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 15:42:30.901018 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.901008 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 15:42:30.901208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.901194 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 15:42:30.901778 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.901754 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 15:42:30.901899 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.901884 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2ljcb\"" Jul 07 15:42:30.910894 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.909782 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c89d7c965-rtqd9"] Jul 07 15:42:30.910894 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.910704 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 15:42:30.925907 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.925884 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-serving-cert\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:30.926007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.925954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-oauth-config\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:30.926007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.925971 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmd7r\" (UniqueName: \"kubernetes.io/projected/49273c59-63a8-4691-9077-6abae82484c5-kube-api-access-cmd7r\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:30.926007 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.926004 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-console-config\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:30.926177 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.926023 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-trusted-ca-bundle\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:30.926177 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.926037 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-oauth-serving-cert\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:30.926177 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:30.926053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-service-ca\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.004238 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.004154 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-66js4" event={"ID":"3d1ee745-d61d-4d6d-bda8-6eb2417bd150","Type":"ContainerStarted","Data":"7205883465d9c4296f72918bfecc04774dd0e36906a6231d131f9312269cb3b8"} Jul 07 15:42:31.005873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.005848 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgwz7" event={"ID":"b0bc8fd0-fce0-499d-a241-2df1853d1f60","Type":"ContainerStarted","Data":"91e2ce2402d1468e4a696f7353b7f58bafdb369bb5b22bd90e96922c0bd996bc"} Jul 07 15:42:31.006005 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.005907 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:31.006005 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.005931 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgwz7" event={"ID":"b0bc8fd0-fce0-499d-a241-2df1853d1f60","Type":"ContainerStarted","Data":"c978480507f48d386aba8f03a98f8f57613caa184413595dbda8be43dd2da5c8"} Jul 07 15:42:31.007242 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.007220 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r7ldp" event={"ID":"7aae2321-d8a2-4a29-a053-84d7d79b1f6a","Type":"ContainerStarted","Data":"439ab0ea67ad04b092223f2ccf27a57b1001cb498c75fbd3a26b248596aacf0a"} Jul 07 15:42:31.008109 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.008091 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" event={"ID":"bed3c459-1a4f-4517-9939-ac95d82c1dc1","Type":"ContainerStarted","Data":"e841cc53d20a2b6c01c1073d6f3bf61d96edb29660281fe69c528457dec0f0db"} Jul 07 15:42:31.027192 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.027167 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-oauth-config\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.027279 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.027200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmd7r\" (UniqueName: \"kubernetes.io/projected/49273c59-63a8-4691-9077-6abae82484c5-kube-api-access-cmd7r\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.027888 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.027514 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-console-config\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.027888 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.027738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-trusted-ca-bundle\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.027888 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.027768 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-oauth-serving-cert\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.027888 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.027827 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-service-ca\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.028162 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.028101 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-serving-cert\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.028576 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.028512 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-oauth-serving-cert\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.028576 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.028569 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-trusted-ca-bundle\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.028813 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.028570 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-console-config\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.028957 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.028921 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-66js4" podStartSLOduration=0.940037436 podStartE2EDuration="6.02890991s" totalImagesPullingTime="5.088872474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:31.027466964 +0000 UTC m=+57.157880110" watchObservedRunningTime="2026-07-07 15:42:31.02890991 +0000 UTC m=+57.159323033" Jul 07 15:42:31.029180 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.029155 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-service-ca\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.029927 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.029908 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-oauth-config\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.030439 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.030423 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-serving-cert\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.037621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.037602 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmd7r\" (UniqueName: \"kubernetes.io/projected/49273c59-63a8-4691-9077-6abae82484c5-kube-api-access-cmd7r\") pod \"console-5c89d7c965-rtqd9\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.050812 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.050775 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mgwz7" podStartSLOduration=1.901877644 podStartE2EDuration="7.050762242s" totalImagesPullingTime="5.148884598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:31.050102932 +0000 UTC m=+57.180516079" watchObservedRunningTime="2026-07-07 15:42:31.050762242 +0000 UTC m=+57.181175388" Jul 07 15:42:31.207401 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.207366 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:31.327948 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:31.327919 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c89d7c965-rtqd9"] Jul 07 15:42:31.330470 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:31.330442 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49273c59_63a8_4691_9077_6abae82484c5.slice/crio-773e25f1409ac77037efcf8d6768851c1aa563163cc2c0a1a0bf2031cc5de5ac WatchSource:0}: Error finding container 773e25f1409ac77037efcf8d6768851c1aa563163cc2c0a1a0bf2031cc5de5ac: Status 404 returned error can't find the container with id 773e25f1409ac77037efcf8d6768851c1aa563163cc2c0a1a0bf2031cc5de5ac Jul 07 15:42:32.014694 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:32.014648 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r7ldp" event={"ID":"7aae2321-d8a2-4a29-a053-84d7d79b1f6a","Type":"ContainerStarted","Data":"65b245aa51f55411d322ef3c2c704ef0cf48bd5c32ae4d6d59d3d74d5e6a5a3a"} Jul 07 15:42:32.016751 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:32.016720 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c89d7c965-rtqd9" event={"ID":"49273c59-63a8-4691-9077-6abae82484c5","Type":"ContainerStarted","Data":"773e25f1409ac77037efcf8d6768851c1aa563163cc2c0a1a0bf2031cc5de5ac"} Jul 07 15:42:33.026254 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:33.026206 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" event={"ID":"bed3c459-1a4f-4517-9939-ac95d82c1dc1","Type":"ContainerStarted","Data":"012cd7429ee149762d391336b5c2c79d6fece79d907d0dfedeb46b2be4834273"} Jul 07 15:42:33.026733 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:33.026300 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" event={"ID":"bed3c459-1a4f-4517-9939-ac95d82c1dc1","Type":"ContainerStarted","Data":"a0db6b0b47d2ab9fe4075e94770b0aea85cdbd50374879bfaa97b655e79d2f08"} Jul 07 15:42:33.047413 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:33.047365 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bk4br" podStartSLOduration=3.469284062 podStartE2EDuration="5.047347593s" totalImagesPullingTime="1.578063531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:33.046644959 +0000 UTC m=+59.177058106" watchObservedRunningTime="2026-07-07 15:42:33.047347593 +0000 UTC m=+59.177760739" Jul 07 15:42:34.632210 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.632184 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d"] Jul 07 15:42:34.642669 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.642649 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.645326 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.645260 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 15:42:34.645326 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.645264 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 15:42:34.645326 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.645316 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mtglq\"" Jul 07 15:42:34.647718 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.647695 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mmfc2"] Jul 07 15:42:34.653273 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.653247 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d"] Jul 07 15:42:34.654072 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.654056 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.657289 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.657273 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 15:42:34.657400 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.657276 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 15:42:34.657400 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.657333 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p7tzj\"" Jul 07 15:42:34.657681 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.657664 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 15:42:34.658020 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.657999 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa73cf8-abd2-44ce-b682-e756427829de-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.658115 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.658048 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw7z7\" (UniqueName: \"kubernetes.io/projected/caa73cf8-abd2-44ce-b682-e756427829de-kube-api-access-lw7z7\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.658115 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.658084 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/caa73cf8-abd2-44ce-b682-e756427829de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.658235 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.658216 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa73cf8-abd2-44ce-b682-e756427829de-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.674971 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.674943 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn"] Jul 07 15:42:34.688032 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.688015 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.690656 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.690623 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 15:42:34.690959 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.690938 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 15:42:34.691336 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.691311 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-497zh\"" Jul 07 15:42:34.691430 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.691362 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 15:42:34.694146 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.694120 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn"] Jul 07 15:42:34.758646 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758616 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/763adb88-1e13-496b-919c-f7aa8c98746d-root\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.758804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758661 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf96a278-8b70-4c98-92f7-06739a72140a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.758804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758707 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa73cf8-abd2-44ce-b682-e756427829de-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.758804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758757 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf96a278-8b70-4c98-92f7-06739a72140a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.758804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-wtmp\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.759039 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758827 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlk6\" (UniqueName: \"kubernetes.io/projected/763adb88-1e13-496b-919c-f7aa8c98746d-kube-api-access-njlk6\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.759039 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758855 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-textfile\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.759039 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758927 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-tls\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.759039 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758961 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/763adb88-1e13-496b-919c-f7aa8c98746d-metrics-client-ca\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.759039 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.758983 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf96a278-8b70-4c98-92f7-06739a72140a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.759039 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa73cf8-abd2-44ce-b682-e756427829de-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.759039 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759029 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.759360 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw7z7\" (UniqueName: \"kubernetes.io/projected/caa73cf8-abd2-44ce-b682-e756427829de-kube-api-access-lw7z7\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.759360 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759085 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/caa73cf8-abd2-44ce-b682-e756427829de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.759360 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759107 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-accelerators-collector-config\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.759360 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759136 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf96a278-8b70-4c98-92f7-06739a72140a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.759360 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759162 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf96a278-8b70-4c98-92f7-06739a72140a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.759360 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/763adb88-1e13-496b-919c-f7aa8c98746d-sys\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.759360 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759341 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpr5\" (UniqueName: \"kubernetes.io/projected/bf96a278-8b70-4c98-92f7-06739a72140a-kube-api-access-jlpr5\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.759743 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:34.759624 2583 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 07 15:42:34.759743 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:34.759718 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa73cf8-abd2-44ce-b682-e756427829de-openshift-state-metrics-tls podName:caa73cf8-abd2-44ce-b682-e756427829de nodeName:}" failed. No retries permitted until 2026-07-07 15:42:35.259698782 +0000 UTC m=+61.390111908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/caa73cf8-abd2-44ce-b682-e756427829de-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-pwh8d" (UID: "caa73cf8-abd2-44ce-b682-e756427829de") : secret "openshift-state-metrics-tls" not found Jul 07 15:42:34.759858 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.759838 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa73cf8-abd2-44ce-b682-e756427829de-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.762392 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.762369 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa73cf8-abd2-44ce-b682-e756427829de-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.772288 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.772261 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw7z7\" (UniqueName: \"kubernetes.io/projected/caa73cf8-abd2-44ce-b682-e756427829de-kube-api-access-lw7z7\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:34.860656 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/763adb88-1e13-496b-919c-f7aa8c98746d-metrics-client-ca\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.860804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860663 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf96a278-8b70-4c98-92f7-06739a72140a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.860804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860696 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.860804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-accelerators-collector-config\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.860804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf96a278-8b70-4c98-92f7-06739a72140a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.861019 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf96a278-8b70-4c98-92f7-06739a72140a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.861019 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860850 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/763adb88-1e13-496b-919c-f7aa8c98746d-sys\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861019 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860878 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpr5\" (UniqueName: \"kubernetes.io/projected/bf96a278-8b70-4c98-92f7-06739a72140a-kube-api-access-jlpr5\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.861019 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/763adb88-1e13-496b-919c-f7aa8c98746d-root\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861019 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf96a278-8b70-4c98-92f7-06739a72140a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.861019 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.860975 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf96a278-8b70-4c98-92f7-06739a72140a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.861411 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861020 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-wtmp\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861411 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861046 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlk6\" (UniqueName: \"kubernetes.io/projected/763adb88-1e13-496b-919c-f7aa8c98746d-kube-api-access-njlk6\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861411 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-textfile\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861411 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861086 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/763adb88-1e13-496b-919c-f7aa8c98746d-sys\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861411 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861112 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-tls\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861411 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861299 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/763adb88-1e13-496b-919c-f7aa8c98746d-metrics-client-ca\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861727 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861422 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf96a278-8b70-4c98-92f7-06739a72140a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.861727 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861486 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf96a278-8b70-4c98-92f7-06739a72140a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.861960 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861889 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-accelerators-collector-config\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861960 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861908 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/763adb88-1e13-496b-919c-f7aa8c98746d-root\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.861960 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861893 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-wtmp\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.862139 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.861957 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf96a278-8b70-4c98-92f7-06739a72140a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.862365 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.862324 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-textfile\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.864018 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.863971 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-tls\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.864289 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.864262 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf96a278-8b70-4c98-92f7-06739a72140a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.864649 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.864625 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf96a278-8b70-4c98-92f7-06739a72140a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.864786 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.864759 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/763adb88-1e13-496b-919c-f7aa8c98746d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.876802 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.876779 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlk6\" (UniqueName: \"kubernetes.io/projected/763adb88-1e13-496b-919c-f7aa8c98746d-kube-api-access-njlk6\") pod \"node-exporter-mmfc2\" (UID: \"763adb88-1e13-496b-919c-f7aa8c98746d\") " pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.877006 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.876988 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpr5\" (UniqueName: \"kubernetes.io/projected/bf96a278-8b70-4c98-92f7-06739a72140a-kube-api-access-jlpr5\") pod \"kube-state-metrics-7c9dbb8b7d-dnsvn\" (UID: \"bf96a278-8b70-4c98-92f7-06739a72140a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:34.943096 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.943033 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzqpt" Jul 07 15:42:34.965801 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.965775 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mmfc2" Jul 07 15:42:34.999414 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:34.999373 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" Jul 07 15:42:35.263966 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.263872 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/caa73cf8-abd2-44ce-b682-e756427829de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:35.266478 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.266454 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/caa73cf8-abd2-44ce-b682-e756427829de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pwh8d\" (UID: \"caa73cf8-abd2-44ce-b682-e756427829de\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:35.527404 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:35.527374 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763adb88_1e13_496b_919c_f7aa8c98746d.slice/crio-cde74d671d9a4a3d7a4597f3eb629160153b2131295034c444a6d9357d3a550b WatchSource:0}: Error finding container cde74d671d9a4a3d7a4597f3eb629160153b2131295034c444a6d9357d3a550b: Status 404 returned error can't find the container with id cde74d671d9a4a3d7a4597f3eb629160153b2131295034c444a6d9357d3a550b Jul 07 15:42:35.556570 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.556532 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" Jul 07 15:42:35.683001 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.682928 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn"] Jul 07 15:42:35.684678 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:35.684637 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf96a278_8b70_4c98_92f7_06739a72140a.slice/crio-28dd3b2f179eb3ffa89c5613185dabcaaed76a319480160da899b50c80461bc6 WatchSource:0}: Error finding container 28dd3b2f179eb3ffa89c5613185dabcaaed76a319480160da899b50c80461bc6: Status 404 returned error can't find the container with id 28dd3b2f179eb3ffa89c5613185dabcaaed76a319480160da899b50c80461bc6 Jul 07 15:42:35.713106 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.713074 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d"] Jul 07 15:42:35.716268 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:35.716238 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa73cf8_abd2_44ce_b682_e756427829de.slice/crio-dcedbb85c5aa0b4f6f2655b881db72d17833c92c1163c836dfce5e12cf6b199d WatchSource:0}: Error finding container dcedbb85c5aa0b4f6f2655b881db72d17833c92c1163c836dfce5e12cf6b199d: Status 404 returned error can't find the container with id dcedbb85c5aa0b4f6f2655b881db72d17833c92c1163c836dfce5e12cf6b199d Jul 07 15:42:35.769865 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.768867 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 15:42:35.772630 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.772610 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.775505 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.775397 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cn6k8\"" Jul 07 15:42:35.775505 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.775398 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 15:42:35.776362 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.775752 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 15:42:35.776362 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.775884 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 15:42:35.776362 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.775960 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 15:42:35.776362 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.776007 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 15:42:35.776362 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.776022 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 15:42:35.776362 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.776271 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 15:42:35.776720 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.776414 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 15:42:35.776720 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.776628 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 15:42:35.792701 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.792590 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 15:42:35.870042 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870042 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870247 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870073 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870247 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870100 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870247 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870147 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870247 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870200 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870247 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-web-config\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870407 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870270 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870407 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870287 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdpw8\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-kube-api-access-hdpw8\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870407 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870309 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-config-out\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870407 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870331 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870407 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870354 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.870407 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.870377 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.971124 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.971077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.971309 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.971132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.971309 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.971169 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.971309 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.971239 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.971309 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.971269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.971309 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.971308 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.971651 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.971332 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-web-config\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.971651 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.971376 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.973491 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.971998 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.973491 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.972051 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdpw8\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-kube-api-access-hdpw8\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.973491 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.972096 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-config-out\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.973491 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.972132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.973491 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.972170 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.973491 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.972203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.973491 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:35.972314 2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 15:42:35.973491 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:35.972370 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-main-tls podName:2213804f-ea04-431e-9cd8-62fb95d6664f nodeName:}" failed. No retries permitted until 2026-07-07 15:42:36.472352585 +0000 UTC m=+62.602765717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f") : secret "alertmanager-main-tls" not found Jul 07 15:42:35.974276 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.974250 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.974475 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.974452 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.974666 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:42:35.974615 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-trusted-ca-bundle podName:2213804f-ea04-431e-9cd8-62fb95d6664f nodeName:}" failed. No retries permitted until 2026-07-07 15:42:36.474596558 +0000 UTC m=+62.605009699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f") : configmap references non-existent config key: ca-bundle.crt Jul 07 15:42:35.974871 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.974824 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.975057 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.975015 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.976027 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.976004 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.976252 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.976228 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.976406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.976383 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-web-config\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.976897 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.976873 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.977043 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.977023 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-config-out\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:35.988582 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:35.988540 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdpw8\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-kube-api-access-hdpw8\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:36.047323 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.047240 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" event={"ID":"caa73cf8-abd2-44ce-b682-e756427829de","Type":"ContainerStarted","Data":"9f58df69de1dd0745f967a8de110c800b2a4f43bbb305e236b740bfca3cd4d16"} Jul 07 15:42:36.047477 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.047334 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" event={"ID":"caa73cf8-abd2-44ce-b682-e756427829de","Type":"ContainerStarted","Data":"66a874e3d5a27b7693d511bc18ba2b57a6b769636b69bcaf5e2c12b0a8eb3c0d"} Jul 07 15:42:36.047477 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.047363 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" event={"ID":"caa73cf8-abd2-44ce-b682-e756427829de","Type":"ContainerStarted","Data":"dcedbb85c5aa0b4f6f2655b881db72d17833c92c1163c836dfce5e12cf6b199d"} Jul 07 15:42:36.049501 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.049466 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r7ldp" event={"ID":"7aae2321-d8a2-4a29-a053-84d7d79b1f6a","Type":"ContainerStarted","Data":"c735e8be134ae5fddf3c27860dde10df4dc7c33433fbbd60b5f3512423457804"} Jul 07 15:42:36.051149 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.051123 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c89d7c965-rtqd9" event={"ID":"49273c59-63a8-4691-9077-6abae82484c5","Type":"ContainerStarted","Data":"98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b"} Jul 07 15:42:36.052513 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.052490 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" event={"ID":"bf96a278-8b70-4c98-92f7-06739a72140a","Type":"ContainerStarted","Data":"28dd3b2f179eb3ffa89c5613185dabcaaed76a319480160da899b50c80461bc6"} Jul 07 15:42:36.054195 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.054174 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mmfc2" event={"ID":"763adb88-1e13-496b-919c-f7aa8c98746d","Type":"ContainerStarted","Data":"cde74d671d9a4a3d7a4597f3eb629160153b2131295034c444a6d9357d3a550b"} Jul 07 15:42:36.075021 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.074971 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r7ldp" podStartSLOduration=2.248341146 podStartE2EDuration="12.07495418s" totalImagesPullingTime="9.826613034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:42:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:36.073986786 +0000 UTC m=+62.204399935" watchObservedRunningTime="2026-07-07 15:42:36.07495418 +0000 UTC m=+62.205367326" Jul 07 15:42:36.097789 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.097736 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c89d7c965-rtqd9" podStartSLOduration=1.902123167 podStartE2EDuration="6.097719547s" totalImagesPullingTime="4.19559638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:36.09689128 +0000 UTC m=+62.227304425" watchObservedRunningTime="2026-07-07 15:42:36.097719547 +0000 UTC m=+62.228132693" Jul 07 15:42:36.477736 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.477701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:36.477736 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.477741 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:36.478716 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.478690 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:36.480096 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.480067 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:36.695613 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.695495 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:42:36.775389 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.775344 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f89874c98-95w2h"] Jul 07 15:42:36.778767 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.778739 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.781533 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.781508 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 15:42:36.781936 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.781798 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fxc6v\"" Jul 07 15:42:36.781936 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.781843 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 15:42:36.782509 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.782169 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 15:42:36.782509 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.782232 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 15:42:36.782509 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.782170 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-co98ra21cdgdv\"" Jul 07 15:42:36.782509 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.782425 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 15:42:36.801390 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.801359 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f89874c98-95w2h"] Jul 07 15:42:36.859259 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.859225 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 15:42:36.880810 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.880777 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.880810 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.880814 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2c34f0d-e299-4a0e-a693-08541fd93d20-metrics-client-ca\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.881036 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.880859 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-tls\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.881036 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.880913 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.881036 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.880956 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.881036 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.880990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.881036 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.881016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-grpc-tls\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.881200 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.881100 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ttbb\" (UniqueName: \"kubernetes.io/projected/b2c34f0d-e299-4a0e-a693-08541fd93d20-kube-api-access-8ttbb\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.982223 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.982131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.982223 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.982172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2c34f0d-e299-4a0e-a693-08541fd93d20-metrics-client-ca\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.982223 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.982214 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-tls\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.982516 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.982256 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.982516 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.982304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.982516 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.982341 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.982516 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.982381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-grpc-tls\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.982516 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.982414 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ttbb\" (UniqueName: \"kubernetes.io/projected/b2c34f0d-e299-4a0e-a693-08541fd93d20-kube-api-access-8ttbb\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.983186 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.983152 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2c34f0d-e299-4a0e-a693-08541fd93d20-metrics-client-ca\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.985341 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.985309 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.985450 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.985309 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.985645 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.985623 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.985802 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.985781 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.985873 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.985790 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-grpc-tls\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.986130 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.986102 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b2c34f0d-e299-4a0e-a693-08541fd93d20-secret-thanos-querier-tls\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:36.993013 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:36.992987 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ttbb\" (UniqueName: \"kubernetes.io/projected/b2c34f0d-e299-4a0e-a693-08541fd93d20-kube-api-access-8ttbb\") pod \"thanos-querier-7f89874c98-95w2h\" (UID: \"b2c34f0d-e299-4a0e-a693-08541fd93d20\") " pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:37.059463 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:37.059424 2583 generic.go:356] "Generic (PLEG): container finished" podID="763adb88-1e13-496b-919c-f7aa8c98746d" containerID="be96e9b6bd0765e2806ef7837dd2a0282f05e8a3d44abeecef3286721552e2d2" exitCode=0 Jul 07 15:42:37.059648 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:37.059520 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mmfc2" event={"ID":"763adb88-1e13-496b-919c-f7aa8c98746d","Type":"ContainerDied","Data":"be96e9b6bd0765e2806ef7837dd2a0282f05e8a3d44abeecef3286721552e2d2"} Jul 07 15:42:37.078981 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:37.078949 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2213804f_ea04_431e_9cd8_62fb95d6664f.slice/crio-1128875b7c67c062b6e50cc8a16f07408a6d2fc2d6327bbda15123417cd9efe4 WatchSource:0}: Error finding container 1128875b7c67c062b6e50cc8a16f07408a6d2fc2d6327bbda15123417cd9efe4: Status 404 returned error can't find the container with id 1128875b7c67c062b6e50cc8a16f07408a6d2fc2d6327bbda15123417cd9efe4 Jul 07 15:42:37.088518 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:37.088464 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:37.515229 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:37.515166 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f89874c98-95w2h"] Jul 07 15:42:37.519749 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:37.519708 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c34f0d_e299_4a0e_a693_08541fd93d20.slice/crio-82825dcd596a6220104779b51ad553d8a8eab48d38e6ad08e5c9f33ba64e6047 WatchSource:0}: Error finding container 82825dcd596a6220104779b51ad553d8a8eab48d38e6ad08e5c9f33ba64e6047: Status 404 returned error can't find the container with id 82825dcd596a6220104779b51ad553d8a8eab48d38e6ad08e5c9f33ba64e6047 Jul 07 15:42:38.066040 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.066005 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mmfc2" event={"ID":"763adb88-1e13-496b-919c-f7aa8c98746d","Type":"ContainerStarted","Data":"703adf8a2954f506adb3583b56d40d046268622ce117e13f3d8c220fe5baa34d"} Jul 07 15:42:38.066516 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.066073 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mmfc2" event={"ID":"763adb88-1e13-496b-919c-f7aa8c98746d","Type":"ContainerStarted","Data":"d587d76fcd531db3a686ef4106becb726420ef484d6a3a4494ce941ec4e35b9f"} Jul 07 15:42:38.068349 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.068314 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" event={"ID":"caa73cf8-abd2-44ce-b682-e756427829de","Type":"ContainerStarted","Data":"16e65fa50de636cf2e54c79dd2d069b5b65d392ad8799c5b2f4b9925cbd46619"} Jul 07 15:42:38.069663 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.069635 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" event={"ID":"b2c34f0d-e299-4a0e-a693-08541fd93d20","Type":"ContainerStarted","Data":"82825dcd596a6220104779b51ad553d8a8eab48d38e6ad08e5c9f33ba64e6047"} Jul 07 15:42:38.071482 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.071404 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerStarted","Data":"1128875b7c67c062b6e50cc8a16f07408a6d2fc2d6327bbda15123417cd9efe4"} Jul 07 15:42:38.074028 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.074003 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" event={"ID":"bf96a278-8b70-4c98-92f7-06739a72140a","Type":"ContainerStarted","Data":"ec6b67b79fa63bda6546e49600aad37ca4f75359a4223aa291a708eedd8f9162"} Jul 07 15:42:38.074112 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.074050 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" event={"ID":"bf96a278-8b70-4c98-92f7-06739a72140a","Type":"ContainerStarted","Data":"56fb928c6dde96aad331308256e2f8a01d1954fafc697755ade16e6653de1380"} Jul 07 15:42:38.074112 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.074072 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" event={"ID":"bf96a278-8b70-4c98-92f7-06739a72140a","Type":"ContainerStarted","Data":"b69b8a7fe15b53a7635587ec83590b341d40f8db59679fb4ac79dfba012f9576"} Jul 07 15:42:38.088652 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.088573 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mmfc2" podStartSLOduration=3.213797628 podStartE2EDuration="4.088538289s" totalImagesPullingTime="808.951868ms" totalInitContainerRuntime="65.788793ms" isStatefulPod=true podCreationTimestamp="2026-07-07 15:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:38.088298651 +0000 UTC m=+64.218711798" watchObservedRunningTime="2026-07-07 15:42:38.088538289 +0000 UTC m=+64.218951440" Jul 07 15:42:38.119578 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.119506 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pwh8d" podStartSLOduration=2.638390342 podStartE2EDuration="4.119481736s" totalImagesPullingTime="1.481091394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:38.118065673 +0000 UTC m=+64.248478818" watchObservedRunningTime="2026-07-07 15:42:38.119481736 +0000 UTC m=+64.249894863" Jul 07 15:42:38.141488 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.141425 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dnsvn" podStartSLOduration=2.512882897 podStartE2EDuration="4.141406724s" totalImagesPullingTime="1.628523827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:38.140073252 +0000 UTC m=+64.270486399" watchObservedRunningTime="2026-07-07 15:42:38.141406724 +0000 UTC m=+64.271819871" Jul 07 15:42:38.954681 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.954646 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5888f59b74-85v4h"] Jul 07 15:42:38.957240 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.957216 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:38.960685 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.960662 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 15:42:38.960807 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.960711 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 15:42:38.961615 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.961596 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 15:42:38.962231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.962110 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gtwmc\"" Jul 07 15:42:38.962231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.962128 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 15:42:38.962231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.962195 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-unpge6gcpu7v\"" Jul 07 15:42:38.970038 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:38.969985 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5888f59b74-85v4h"] Jul 07 15:42:39.004397 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.004361 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2b1cbc78-5d52-4726-9b93-9bf60debadd9-audit-log\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.004589 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.004419 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2b1cbc78-5d52-4726-9b93-9bf60debadd9-metrics-server-audit-profiles\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.004589 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.004456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2b1cbc78-5d52-4726-9b93-9bf60debadd9-secret-metrics-server-client-certs\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.004589 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.004486 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzt26\" (UniqueName: \"kubernetes.io/projected/2b1cbc78-5d52-4726-9b93-9bf60debadd9-kube-api-access-wzt26\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.004734 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.004585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2b1cbc78-5d52-4726-9b93-9bf60debadd9-secret-metrics-server-tls\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.004734 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.004611 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b1cbc78-5d52-4726-9b93-9bf60debadd9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.004734 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.004671 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2b1cbc78-5d52-4726-9b93-9bf60debadd9-client-ca-bundle\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.079609 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.079565 2583 generic.go:356] "Generic (PLEG): container finished" podID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerID="1546bb9acc29b22a9224b20746a34ab91731d5cfce3f17a9de9ace8eb9e82448" exitCode=0 Jul 07 15:42:39.080025 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.079629 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerDied","Data":"1546bb9acc29b22a9224b20746a34ab91731d5cfce3f17a9de9ace8eb9e82448"} Jul 07 15:42:39.105862 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.105818 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzt26\" (UniqueName: \"kubernetes.io/projected/2b1cbc78-5d52-4726-9b93-9bf60debadd9-kube-api-access-wzt26\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.106366 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.106334 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2b1cbc78-5d52-4726-9b93-9bf60debadd9-secret-metrics-server-tls\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.106494 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.106378 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b1cbc78-5d52-4726-9b93-9bf60debadd9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.107451 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.106575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2b1cbc78-5d52-4726-9b93-9bf60debadd9-client-ca-bundle\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.107451 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.106950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2b1cbc78-5d52-4726-9b93-9bf60debadd9-audit-log\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.107451 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.107058 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2b1cbc78-5d52-4726-9b93-9bf60debadd9-metrics-server-audit-profiles\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.107451 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.107137 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2b1cbc78-5d52-4726-9b93-9bf60debadd9-secret-metrics-server-client-certs\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.107451 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.107316 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b1cbc78-5d52-4726-9b93-9bf60debadd9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.107451 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.107406 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2b1cbc78-5d52-4726-9b93-9bf60debadd9-audit-log\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.108135 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.108094 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2b1cbc78-5d52-4726-9b93-9bf60debadd9-metrics-server-audit-profiles\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.109421 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.109396 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2b1cbc78-5d52-4726-9b93-9bf60debadd9-secret-metrics-server-tls\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.109701 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.109665 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2b1cbc78-5d52-4726-9b93-9bf60debadd9-client-ca-bundle\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.109970 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.109946 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2b1cbc78-5d52-4726-9b93-9bf60debadd9-secret-metrics-server-client-certs\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.126509 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.126477 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzt26\" (UniqueName: \"kubernetes.io/projected/2b1cbc78-5d52-4726-9b93-9bf60debadd9-kube-api-access-wzt26\") pod \"metrics-server-5888f59b74-85v4h\" (UID: \"2b1cbc78-5d52-4726-9b93-9bf60debadd9\") " pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.268651 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.268540 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:39.534997 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.534967 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5888f59b74-85v4h"] Jul 07 15:42:39.581575 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:39.581533 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1cbc78_5d52_4726_9b93_9bf60debadd9.slice/crio-75fb51db6af3791201bc8e325c24b99195251f7e64cdad8265b624513acce58d WatchSource:0}: Error finding container 75fb51db6af3791201bc8e325c24b99195251f7e64cdad8265b624513acce58d: Status 404 returned error can't find the container with id 75fb51db6af3791201bc8e325c24b99195251f7e64cdad8265b624513acce58d Jul 07 15:42:39.665404 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.665368 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c89d7c965-rtqd9"] Jul 07 15:42:39.690124 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.690089 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t"] Jul 07 15:42:39.693075 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.693054 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:39.696387 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.696365 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 15:42:39.696567 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.696385 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-97smz\"" Jul 07 15:42:39.712029 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.711985 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw8th\" (UniqueName: \"kubernetes.io/projected/940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9-kube-api-access-gw8th\") pod \"monitoring-plugin-d5dcff7cf-f7d5t\" (UID: \"940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:39.712182 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.712159 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f7d5t\" (UID: \"940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:39.720037 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.720012 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t"] Jul 07 15:42:39.727248 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.727204 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4d74674-qvm6x"] Jul 07 15:42:39.729667 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.729646 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.746655 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.746621 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4d74674-qvm6x"] Jul 07 15:42:39.813644 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.813612 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-service-ca\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.813783 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.813647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-serving-cert\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.813783 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.813681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f7d5t\" (UID: \"940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:39.813783 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.813729 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp4ld\" (UniqueName: \"kubernetes.io/projected/5243c61f-74fe-49d0-859e-780879e50208-kube-api-access-qp4ld\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.813944 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.813781 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-console-config\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.813944 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.813827 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-trusted-ca-bundle\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.813944 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.813872 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw8th\" (UniqueName: \"kubernetes.io/projected/940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9-kube-api-access-gw8th\") pod \"monitoring-plugin-d5dcff7cf-f7d5t\" (UID: \"940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:39.813944 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.813893 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-oauth-config\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.813944 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.813938 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-oauth-serving-cert\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.816056 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.816037 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f7d5t\" (UID: \"940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:39.825998 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.825963 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw8th\" (UniqueName: \"kubernetes.io/projected/940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9-kube-api-access-gw8th\") pod \"monitoring-plugin-d5dcff7cf-f7d5t\" (UID: \"940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:39.915369 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.915333 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-service-ca\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.915369 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.915370 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-serving-cert\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.915622 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.915492 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp4ld\" (UniqueName: \"kubernetes.io/projected/5243c61f-74fe-49d0-859e-780879e50208-kube-api-access-qp4ld\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.915622 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.915535 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-console-config\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.915718 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.915621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-trusted-ca-bundle\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.915718 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.915668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-oauth-config\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.915907 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.915882 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-oauth-serving-cert\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.916246 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.916194 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-service-ca\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.916348 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.916337 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-console-config\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.916622 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.916596 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-oauth-serving-cert\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.916699 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.916639 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-trusted-ca-bundle\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.918540 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.918517 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-oauth-config\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.918643 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.918629 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-serving-cert\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:39.928724 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:39.928693 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp4ld\" (UniqueName: \"kubernetes.io/projected/5243c61f-74fe-49d0-859e-780879e50208-kube-api-access-qp4ld\") pod \"console-6c4d74674-qvm6x\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:40.004316 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.004280 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:40.041269 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.041229 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:40.087608 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.087507 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" event={"ID":"2b1cbc78-5d52-4726-9b93-9bf60debadd9","Type":"ContainerStarted","Data":"75fb51db6af3791201bc8e325c24b99195251f7e64cdad8265b624513acce58d"} Jul 07 15:42:40.092189 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.092150 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" event={"ID":"b2c34f0d-e299-4a0e-a693-08541fd93d20","Type":"ContainerStarted","Data":"d881656a3c62768ba5ec5a957a1612229ea7eb07f4534f43150f9428aed363fc"} Jul 07 15:42:40.092340 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.092239 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" event={"ID":"b2c34f0d-e299-4a0e-a693-08541fd93d20","Type":"ContainerStarted","Data":"23dc5c78882515821c309ac937715cf65a8d98ed495daedcf3287e0a88797fca"} Jul 07 15:42:40.092340 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.092283 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" event={"ID":"b2c34f0d-e299-4a0e-a693-08541fd93d20","Type":"ContainerStarted","Data":"f52233e4afdca67e0ddf7b003ff232953b46c133de09cc51c8307703d457e070"} Jul 07 15:42:40.173614 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.173511 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t"] Jul 07 15:42:40.181632 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:40.181599 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940fa5fe_ff38_4d5f_ab3f_2b0cc0cc2ef9.slice/crio-04767b035e7ebd37fdde58dcd01edd227bbc9bd600757c5f281ab5c71b2b8693 WatchSource:0}: Error finding container 04767b035e7ebd37fdde58dcd01edd227bbc9bd600757c5f281ab5c71b2b8693: Status 404 returned error can't find the container with id 04767b035e7ebd37fdde58dcd01edd227bbc9bd600757c5f281ab5c71b2b8693 Jul 07 15:42:40.219329 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.219200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:40.222140 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.222117 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 15:42:40.226280 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.226205 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4d74674-qvm6x"] Jul 07 15:42:40.229236 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:40.229209 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5243c61f_74fe_49d0_859e_780879e50208.slice/crio-80404fbc77f31458b1c258bab99740fa5d856313e16f644ed0a2f6e4f07c5a65 WatchSource:0}: Error finding container 80404fbc77f31458b1c258bab99740fa5d856313e16f644ed0a2f6e4f07c5a65: Status 404 returned error can't find the container with id 80404fbc77f31458b1c258bab99740fa5d856313e16f644ed0a2f6e4f07c5a65 Jul 07 15:42:40.233182 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.233157 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa39c3a5-6b00-4d1d-b712-587e31db6d3d-metrics-certs\") pod \"network-metrics-daemon-w4chw\" (UID: \"aa39c3a5-6b00-4d1d-b712-587e31db6d3d\") " pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:40.320089 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.319796 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:40.320089 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.319882 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:40.322961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.322711 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 15:42:40.322961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.322745 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 15:42:40.332789 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.332766 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35748986-3b58-45b7-a1e0-a77c803c1ad3-original-pull-secret\") pod \"global-pull-secret-syncer-mpk6m\" (UID: \"35748986-3b58-45b7-a1e0-a77c803c1ad3\") " pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:40.333344 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.333327 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 15:42:40.343799 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.343694 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzsg\" (UniqueName: \"kubernetes.io/projected/4f5b73dd-893c-4b4d-8b61-512994aaaa9d-kube-api-access-zfzsg\") pod \"network-check-target-xkxc8\" (UID: \"4f5b73dd-893c-4b4d-8b61-512994aaaa9d\") " pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:40.503226 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.503157 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpk6m" Jul 07 15:42:40.511604 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.511349 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k55hz\"" Jul 07 15:42:40.515628 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.515586 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t2q2r\"" Jul 07 15:42:40.518890 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.518869 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:40.524201 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.524172 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4chw" Jul 07 15:42:40.916728 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.916653 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mpk6m"] Jul 07 15:42:40.947266 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.947240 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xkxc8"] Jul 07 15:42:40.955207 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:40.955154 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w4chw"] Jul 07 15:42:41.020440 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.020363 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mgwz7" Jul 07 15:42:41.097842 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.097812 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mpk6m" event={"ID":"35748986-3b58-45b7-a1e0-a77c803c1ad3","Type":"ContainerStarted","Data":"82d8a0481131c20720e7f05384990d4b972f3db38fa576b7ffcabf867644efa3"} Jul 07 15:42:41.099276 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.099241 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" event={"ID":"940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9","Type":"ContainerStarted","Data":"04767b035e7ebd37fdde58dcd01edd227bbc9bd600757c5f281ab5c71b2b8693"} Jul 07 15:42:41.101167 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.101141 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4d74674-qvm6x" event={"ID":"5243c61f-74fe-49d0-859e-780879e50208","Type":"ContainerStarted","Data":"1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430"} Jul 07 15:42:41.101279 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.101188 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4d74674-qvm6x" event={"ID":"5243c61f-74fe-49d0-859e-780879e50208","Type":"ContainerStarted","Data":"80404fbc77f31458b1c258bab99740fa5d856313e16f644ed0a2f6e4f07c5a65"} Jul 07 15:42:41.103705 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.103677 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerStarted","Data":"1c7ef4d1309f3c9b516b4cfdffa271217a1f94146006754b27fc2243cc74e670"} Jul 07 15:42:41.103829 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.103737 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerStarted","Data":"2fd200afa30321071c9b30602060e635484602ad39cb7d4d8396dd295cd9009d"} Jul 07 15:42:41.133320 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.133238 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c4d74674-qvm6x" podStartSLOduration=2.133217791 podStartE2EDuration="2.133217791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:41.133031903 +0000 UTC m=+67.263445048" watchObservedRunningTime="2026-07-07 15:42:41.133217791 +0000 UTC m=+67.263630940" Jul 07 15:42:41.195654 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.195563 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 15:42:41.211885 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.211849 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:42:41.212059 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.211979 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.215302 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.215274 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 15:42:41.215566 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.215525 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 15:42:41.215676 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.215572 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 15:42:41.215676 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.215618 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1qdbp9vrrcckc\"" Jul 07 15:42:41.217450 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.217311 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 15:42:41.217450 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.217335 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 15:42:41.217775 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.217477 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 15:42:41.217775 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.217484 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 15:42:41.217775 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.217587 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 15:42:41.217775 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.217632 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 15:42:41.217775 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.217648 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 15:42:41.220175 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.220144 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 15:42:41.223333 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.221925 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 15:42:41.223333 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.222398 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 15:42:41.223333 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.222405 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-87q8p\"" Jul 07 15:42:41.226258 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.226173 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 15:42:41.227847 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.227636 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 15:42:41.330332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330279 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330312 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330337 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330332 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330360 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-config-out\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330395 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330427 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330459 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330590 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-config\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330610 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330648 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330671 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330712 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2k9\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-kube-api-access-ht2k9\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330737 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-web-config\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.330840 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330775 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.331412 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330799 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.331412 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.330838 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.432852 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.432818 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.432852 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.432857 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.432893 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.432924 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.432952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.432975 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433001 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-config-out\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433061 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433047 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433083 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433262 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-config\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433287 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433333 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433397 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2k9\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-kube-api-access-ht2k9\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.433921 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.433419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-web-config\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.434720 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.434316 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.434720 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.434408 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.436068 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.435985 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.437175 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.436570 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.437175 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.437063 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.437503 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.437477 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.437607 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.437536 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-web-config\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.439388 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.438170 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.439388 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.438802 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.440097 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.439533 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.440097 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.439638 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.440097 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.440027 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.440296 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.440271 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.440350 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.440313 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.440501 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.440480 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-config\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.440714 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.440690 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-config-out\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.441027 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.441010 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.441973 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.441953 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.442258 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.442244 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.446266 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.446249 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2k9\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-kube-api-access-ht2k9\") pod \"prometheus-k8s-0\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.495491 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:41.495460 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa39c3a5_6b00_4d1d_b712_587e31db6d3d.slice/crio-f6f3fb19b97d3ee67dec98583b740ebfc2c94d764773b631167b01b4b7251793 WatchSource:0}: Error finding container f6f3fb19b97d3ee67dec98583b740ebfc2c94d764773b631167b01b4b7251793: Status 404 returned error can't find the container with id f6f3fb19b97d3ee67dec98583b740ebfc2c94d764773b631167b01b4b7251793 Jul 07 15:42:41.496263 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:41.496229 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5b73dd_893c_4b4d_8b61_512994aaaa9d.slice/crio-31e958bf7a7783db7ed3014015872431058b7c288fba06ca6b6b0bcf33ce6aa8 WatchSource:0}: Error finding container 31e958bf7a7783db7ed3014015872431058b7c288fba06ca6b6b0bcf33ce6aa8: Status 404 returned error can't find the container with id 31e958bf7a7783db7ed3014015872431058b7c288fba06ca6b6b0bcf33ce6aa8 Jul 07 15:42:41.529501 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.529471 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:41.707167 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:41.707097 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 15:42:41.888771 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:42:41.888704 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31afd01a_5b8a_440d_afb1_d1f06be1326b.slice/crio-6ebe3fdf4a2c099c8d9ed62569c5ccb1883cd1c1111dc74786307fdc22a0fe53 WatchSource:0}: Error finding container 6ebe3fdf4a2c099c8d9ed62569c5ccb1883cd1c1111dc74786307fdc22a0fe53: Status 404 returned error can't find the container with id 6ebe3fdf4a2c099c8d9ed62569c5ccb1883cd1c1111dc74786307fdc22a0fe53 Jul 07 15:42:42.123952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:42.123883 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerStarted","Data":"0c8f213ac7420d45a002af90718e0b29e8137cd12413fdaa4dfe9c6d038888b2"} Jul 07 15:42:42.124384 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:42.124002 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerStarted","Data":"ad2bad1f14938d23a730d4e1d850b69adf15b6aa5403ab4caad3e8a80c1264f6"} Jul 07 15:42:42.124384 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:42.124047 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerStarted","Data":"a7b870f3a39cfc8278c8a62217cf57b5ba7c953dbe71a13b9553f3cea57520e6"} Jul 07 15:42:42.125782 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:42.125753 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" event={"ID":"2b1cbc78-5d52-4726-9b93-9bf60debadd9","Type":"ContainerStarted","Data":"42a125d5016eb9a9b90302c3395154d47a85051460014551a5fd055a3c741fab"} Jul 07 15:42:42.128781 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:42.128756 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerStarted","Data":"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e"} Jul 07 15:42:42.128863 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:42.128825 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerStarted","Data":"6ebe3fdf4a2c099c8d9ed62569c5ccb1883cd1c1111dc74786307fdc22a0fe53"} Jul 07 15:42:42.131759 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:42.131654 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w4chw" event={"ID":"aa39c3a5-6b00-4d1d-b712-587e31db6d3d","Type":"ContainerStarted","Data":"f6f3fb19b97d3ee67dec98583b740ebfc2c94d764773b631167b01b4b7251793"} Jul 07 15:42:42.134356 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:42.134312 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xkxc8" event={"ID":"4f5b73dd-893c-4b4d-8b61-512994aaaa9d","Type":"ContainerStarted","Data":"31e958bf7a7783db7ed3014015872431058b7c288fba06ca6b6b0bcf33ce6aa8"} Jul 07 15:42:42.181758 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:42.181508 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" podStartSLOduration=1.85348434 podStartE2EDuration="4.181487464s" totalImagesPullingTime="2.328003124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:42.147794115 +0000 UTC m=+68.278207261" watchObservedRunningTime="2026-07-07 15:42:42.181487464 +0000 UTC m=+68.311900610" Jul 07 15:42:43.143008 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:43.142957 2583 generic.go:356] "Generic (PLEG): container finished" podID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerID="0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e" exitCode=0 Jul 07 15:42:43.143479 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:43.143091 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerDied","Data":"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e"} Jul 07 15:42:44.152799 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.152756 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" event={"ID":"b2c34f0d-e299-4a0e-a693-08541fd93d20","Type":"ContainerStarted","Data":"ff6383f06c9e13aec8c5002e98a4de13e192daaa463e3e90d17cefaf26584b1a"} Jul 07 15:42:44.153169 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.152874 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" event={"ID":"b2c34f0d-e299-4a0e-a693-08541fd93d20","Type":"ContainerStarted","Data":"636fde6fc7ceb9b46da01bd9ca9be8cef9d71e12885bfebbb62562a1bca660ad"} Jul 07 15:42:44.153169 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.152922 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:44.153169 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.152957 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" event={"ID":"b2c34f0d-e299-4a0e-a693-08541fd93d20","Type":"ContainerStarted","Data":"aff8102233635c59d0f6828a00c23f8e414dfa274f21db74765e2ca453f25ae7"} Jul 07 15:42:44.158373 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.158228 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerStarted","Data":"188de352fc83e99e271a2a7b8c02fa8efa880346dbc5dd0e2510efe97bd73783"} Jul 07 15:42:44.159918 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.159895 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" Jul 07 15:42:44.161327 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.161004 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" event={"ID":"940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9","Type":"ContainerStarted","Data":"30fda10d9f52d92a602544c4591933374707a2a0688d43b9207443d8ab92b1a1"} Jul 07 15:42:44.161327 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.161159 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:44.163880 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.163857 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w4chw" event={"ID":"aa39c3a5-6b00-4d1d-b712-587e31db6d3d","Type":"ContainerStarted","Data":"17268a64d8d124effc59326fd728cbfc5b762495152dd3813fc6989f1a4b6930"} Jul 07 15:42:44.163984 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.163902 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w4chw" event={"ID":"aa39c3a5-6b00-4d1d-b712-587e31db6d3d","Type":"ContainerStarted","Data":"d1f75786b58110ca0bb5c7826149665bc50e3d3053c808ac2aeefebab020ba23"} Jul 07 15:42:44.167615 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.167594 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" Jul 07 15:42:44.187592 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.187525 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f89874c98-95w2h" podStartSLOduration=2.354058297 podStartE2EDuration="8.187506071s" totalImagesPullingTime="5.833447774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:44.184948922 +0000 UTC m=+70.315362069" watchObservedRunningTime="2026-07-07 15:42:44.187506071 +0000 UTC m=+70.317919238" Jul 07 15:42:44.250075 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.248606 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7d5t" podStartSLOduration=1.865242418 podStartE2EDuration="5.248589205s" totalImagesPullingTime="3.383346787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:44.247837399 +0000 UTC m=+70.378250545" watchObservedRunningTime="2026-07-07 15:42:44.248589205 +0000 UTC m=+70.379002353" Jul 07 15:42:44.323362 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.323091 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.961517458 podStartE2EDuration="9.323068739s" totalImagesPullingTime="4.34113561s" totalInitContainerRuntime="20.415671ms" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:44.321165823 +0000 UTC m=+70.451578982" watchObservedRunningTime="2026-07-07 15:42:44.323068739 +0000 UTC m=+70.453481886" Jul 07 15:42:44.343820 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:44.343467 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w4chw" podStartSLOduration=68.272414726 podStartE2EDuration="1m10.343443248s" totalImagesPullingTime="2.071028522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:44.342463455 +0000 UTC m=+70.472876603" watchObservedRunningTime="2026-07-07 15:42:44.343443248 +0000 UTC m=+70.473856394" Jul 07 15:42:49.196255 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.196201 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mpk6m" event={"ID":"35748986-3b58-45b7-a1e0-a77c803c1ad3","Type":"ContainerStarted","Data":"2ff3d6391b1da8c89e66901321136a81b79fe6c376f430daa2ff9a03983b8836"} Jul 07 15:42:49.199273 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.199246 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerStarted","Data":"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1"} Jul 07 15:42:49.199389 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.199312 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerStarted","Data":"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353"} Jul 07 15:42:49.199389 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.199353 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerStarted","Data":"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5"} Jul 07 15:42:49.199457 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.199394 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerStarted","Data":"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3"} Jul 07 15:42:49.199457 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.199423 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerStarted","Data":"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8"} Jul 07 15:42:49.199457 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.199448 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerStarted","Data":"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33"} Jul 07 15:42:49.200583 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.200541 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xkxc8" event={"ID":"4f5b73dd-893c-4b4d-8b61-512994aaaa9d","Type":"ContainerStarted","Data":"44ecb57b2ddc415f4380708a72409a8c18171ca41a6ec3b346f0e10ee2189bc9"} Jul 07 15:42:49.200693 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.200678 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:42:49.215659 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.215614 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mpk6m" podStartSLOduration=66.809822066 podStartE2EDuration="1m14.215597263s" totalImagesPullingTime="7.405775197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:49.214216436 +0000 UTC m=+75.344629582" watchObservedRunningTime="2026-07-07 15:42:49.215597263 +0000 UTC m=+75.346010409" Jul 07 15:42:49.231639 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.231595 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xkxc8" podStartSLOduration=67.406994221 podStartE2EDuration="1m14.231580715s" totalImagesPullingTime="6.824586494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:49.230975611 +0000 UTC m=+75.361388769" watchObservedRunningTime="2026-07-07 15:42:49.231580715 +0000 UTC m=+75.361993862" Jul 07 15:42:49.264191 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:49.264121 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.04233106 podStartE2EDuration="8.264099577s" totalImagesPullingTime="6.199165776s" totalInitContainerRuntime="22.602741ms" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:49.261103874 +0000 UTC m=+75.391517022" watchObservedRunningTime="2026-07-07 15:42:49.264099577 +0000 UTC m=+75.394512729" Jul 07 15:42:50.041731 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:50.041685 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:50.041907 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:50.041785 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:50.046645 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:50.046622 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:50.208527 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:50.208499 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:42:51.530435 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:51.530398 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:42:59.269280 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:59.269145 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:42:59.269280 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:42:59.269211 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:43:04.690562 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:04.690477 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5c89d7c965-rtqd9" podUID="49273c59-63a8-4691-9077-6abae82484c5" containerName="console" containerID="cri-o://98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b" gracePeriod=15 Jul 07 15:43:04.932121 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:04.932096 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:43:05.052682 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.052592 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-oauth-config\") pod \"49273c59-63a8-4691-9077-6abae82484c5\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " Jul 07 15:43:05.052682 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.052666 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-console-config\") pod \"49273c59-63a8-4691-9077-6abae82484c5\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " Jul 07 15:43:05.052877 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.052822 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-serving-cert\") pod \"49273c59-63a8-4691-9077-6abae82484c5\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " Jul 07 15:43:05.052877 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.052871 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-oauth-serving-cert\") pod \"49273c59-63a8-4691-9077-6abae82484c5\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " Jul 07 15:43:05.052967 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.052901 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmd7r\" (UniqueName: \"kubernetes.io/projected/49273c59-63a8-4691-9077-6abae82484c5-kube-api-access-cmd7r\") pod \"49273c59-63a8-4691-9077-6abae82484c5\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " Jul 07 15:43:05.052967 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.052916 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-trusted-ca-bundle\") pod \"49273c59-63a8-4691-9077-6abae82484c5\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " Jul 07 15:43:05.052967 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.052952 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-service-ca\") pod \"49273c59-63a8-4691-9077-6abae82484c5\" (UID: \"49273c59-63a8-4691-9077-6abae82484c5\") " Jul 07 15:43:05.053122 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.053015 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-console-config" (OuterVolumeSpecName: "console-config") pod "49273c59-63a8-4691-9077-6abae82484c5" (UID: "49273c59-63a8-4691-9077-6abae82484c5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:05.053344 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.053321 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-console-config\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:05.053495 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.053394 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "49273c59-63a8-4691-9077-6abae82484c5" (UID: "49273c59-63a8-4691-9077-6abae82484c5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:05.053495 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.053414 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-service-ca" (OuterVolumeSpecName: "service-ca") pod "49273c59-63a8-4691-9077-6abae82484c5" (UID: "49273c59-63a8-4691-9077-6abae82484c5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:05.053648 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.053566 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "49273c59-63a8-4691-9077-6abae82484c5" (UID: "49273c59-63a8-4691-9077-6abae82484c5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:05.055019 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.054993 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "49273c59-63a8-4691-9077-6abae82484c5" (UID: "49273c59-63a8-4691-9077-6abae82484c5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:05.055134 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.055103 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49273c59-63a8-4691-9077-6abae82484c5-kube-api-access-cmd7r" (OuterVolumeSpecName: "kube-api-access-cmd7r") pod "49273c59-63a8-4691-9077-6abae82484c5" (UID: "49273c59-63a8-4691-9077-6abae82484c5"). InnerVolumeSpecName "kube-api-access-cmd7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:43:05.055134 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.055117 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "49273c59-63a8-4691-9077-6abae82484c5" (UID: "49273c59-63a8-4691-9077-6abae82484c5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:05.154339 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.154286 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-oauth-config\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:05.154339 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.154330 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49273c59-63a8-4691-9077-6abae82484c5-console-serving-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:05.154339 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.154340 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-oauth-serving-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:05.154339 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.154349 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmd7r\" (UniqueName: \"kubernetes.io/projected/49273c59-63a8-4691-9077-6abae82484c5-kube-api-access-cmd7r\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:05.154339 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.154358 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-trusted-ca-bundle\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:05.154681 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.154367 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49273c59-63a8-4691-9077-6abae82484c5-service-ca\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:05.272231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.272197 2583 generic.go:356] "Generic (PLEG): container finished" podID="49273c59-63a8-4691-9077-6abae82484c5" containerID="98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b" exitCode=0 Jul 07 15:43:05.272380 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.272240 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c89d7c965-rtqd9" Jul 07 15:43:05.272380 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.272277 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c89d7c965-rtqd9" event={"ID":"49273c59-63a8-4691-9077-6abae82484c5","Type":"ContainerDied","Data":"98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b"} Jul 07 15:43:05.272380 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.272328 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c89d7c965-rtqd9" event={"ID":"49273c59-63a8-4691-9077-6abae82484c5","Type":"ContainerDied","Data":"773e25f1409ac77037efcf8d6768851c1aa563163cc2c0a1a0bf2031cc5de5ac"} Jul 07 15:43:05.272380 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.272361 2583 scope.go:117] "RemoveContainer" containerID="98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b" Jul 07 15:43:05.281136 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.281114 2583 scope.go:117] "RemoveContainer" containerID="98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b" Jul 07 15:43:05.281422 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:43:05.281400 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b\": container with ID starting with 98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b not found: ID does not exist" containerID="98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b" Jul 07 15:43:05.281485 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.281431 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b"} err="failed to get container status \"98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b\": rpc error: code = NotFound desc = could not find container \"98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b\": container with ID starting with 98802182a2a773191b0b3656b9bdf3f1c2ea2bc8236c8ab3a5770221b846e26b not found: ID does not exist" Jul 07 15:43:05.295115 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.295077 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c89d7c965-rtqd9"] Jul 07 15:43:05.299880 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:05.299860 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c89d7c965-rtqd9"] Jul 07 15:43:06.800497 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:06.800447 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49273c59-63a8-4691-9077-6abae82484c5" path="/var/lib/kubelet/pods/49273c59-63a8-4691-9077-6abae82484c5/volumes" Jul 07 15:43:19.274104 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:19.274076 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:43:19.277736 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:19.277712 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5888f59b74-85v4h" Jul 07 15:43:20.207694 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:20.207667 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xkxc8" Jul 07 15:43:41.529910 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:41.529878 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:43:41.550504 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:41.550479 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:43:42.441710 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:42.441678 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:43:55.305263 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.305229 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 15:43:55.305752 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.305535 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="alertmanager" containerID="cri-o://2fd200afa30321071c9b30602060e635484602ad39cb7d4d8396dd295cd9009d" gracePeriod=120 Jul 07 15:43:55.305752 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.305583 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy-metric" containerID="cri-o://0c8f213ac7420d45a002af90718e0b29e8137cd12413fdaa4dfe9c6d038888b2" gracePeriod=120 Jul 07 15:43:55.305752 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.305651 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="prom-label-proxy" containerID="cri-o://188de352fc83e99e271a2a7b8c02fa8efa880346dbc5dd0e2510efe97bd73783" gracePeriod=120 Jul 07 15:43:55.305752 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.305659 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy-web" containerID="cri-o://a7b870f3a39cfc8278c8a62217cf57b5ba7c953dbe71a13b9553f3cea57520e6" gracePeriod=120 Jul 07 15:43:55.305752 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.305671 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy" containerID="cri-o://ad2bad1f14938d23a730d4e1d850b69adf15b6aa5403ab4caad3e8a80c1264f6" gracePeriod=120 Jul 07 15:43:55.305752 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.305677 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="config-reloader" containerID="cri-o://1c7ef4d1309f3c9b516b4cfdffa271217a1f94146006754b27fc2243cc74e670" gracePeriod=120 Jul 07 15:43:55.479456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.479428 2583 generic.go:356] "Generic (PLEG): container finished" podID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerID="188de352fc83e99e271a2a7b8c02fa8efa880346dbc5dd0e2510efe97bd73783" exitCode=0 Jul 07 15:43:55.479456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.479453 2583 generic.go:356] "Generic (PLEG): container finished" podID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerID="ad2bad1f14938d23a730d4e1d850b69adf15b6aa5403ab4caad3e8a80c1264f6" exitCode=0 Jul 07 15:43:55.479456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.479461 2583 generic.go:356] "Generic (PLEG): container finished" podID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerID="1c7ef4d1309f3c9b516b4cfdffa271217a1f94146006754b27fc2243cc74e670" exitCode=0 Jul 07 15:43:55.479721 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.479469 2583 generic.go:356] "Generic (PLEG): container finished" podID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerID="2fd200afa30321071c9b30602060e635484602ad39cb7d4d8396dd295cd9009d" exitCode=0 Jul 07 15:43:55.479721 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.479499 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerDied","Data":"188de352fc83e99e271a2a7b8c02fa8efa880346dbc5dd0e2510efe97bd73783"} Jul 07 15:43:55.479721 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.479583 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerDied","Data":"ad2bad1f14938d23a730d4e1d850b69adf15b6aa5403ab4caad3e8a80c1264f6"} Jul 07 15:43:55.479721 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.479628 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerDied","Data":"1c7ef4d1309f3c9b516b4cfdffa271217a1f94146006754b27fc2243cc74e670"} Jul 07 15:43:55.479721 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:55.479664 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerDied","Data":"2fd200afa30321071c9b30602060e635484602ad39cb7d4d8396dd295cd9009d"} Jul 07 15:43:56.488606 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.488572 2583 generic.go:356] "Generic (PLEG): container finished" podID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerID="0c8f213ac7420d45a002af90718e0b29e8137cd12413fdaa4dfe9c6d038888b2" exitCode=0 Jul 07 15:43:56.488606 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.488602 2583 generic.go:356] "Generic (PLEG): container finished" podID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerID="a7b870f3a39cfc8278c8a62217cf57b5ba7c953dbe71a13b9553f3cea57520e6" exitCode=0 Jul 07 15:43:56.488974 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.488580 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerDied","Data":"0c8f213ac7420d45a002af90718e0b29e8137cd12413fdaa4dfe9c6d038888b2"} Jul 07 15:43:56.488974 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.488704 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerDied","Data":"a7b870f3a39cfc8278c8a62217cf57b5ba7c953dbe71a13b9553f3cea57520e6"} Jul 07 15:43:56.542702 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.542682 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:56.675854 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.675764 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-config-volume\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.675854 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.675804 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-main-tls\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.675854 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.675836 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-metrics-client-ca\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676111 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.675863 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-tls-assets\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676111 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.675904 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-web-config\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676111 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.675940 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676111 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.675976 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-cluster-tls-config\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676111 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.676020 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-trusted-ca-bundle\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676111 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.676053 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-web\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676111 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.676099 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676439 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.676129 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-config-out\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676439 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.676172 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdpw8\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-kube-api-access-hdpw8\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676439 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.676213 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-main-db\") pod \"2213804f-ea04-431e-9cd8-62fb95d6664f\" (UID: \"2213804f-ea04-431e-9cd8-62fb95d6664f\") " Jul 07 15:43:56.676439 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.676206 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:56.676669 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.676448 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-metrics-client-ca\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.676987 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.676763 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:43:56.679226 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.679157 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:43:56.679226 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.679158 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:56.679395 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.679304 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-config-volume" (OuterVolumeSpecName: "config-volume") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:56.679395 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.679305 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-kube-api-access-hdpw8" (OuterVolumeSpecName: "kube-api-access-hdpw8") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "kube-api-access-hdpw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:43:56.679522 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.679489 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-config-out" (OuterVolumeSpecName: "config-out") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:43:56.679657 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.679634 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:56.679729 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.679683 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:56.679800 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.679782 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:56.681334 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.681303 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:56.684788 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.684754 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:56.689267 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.689245 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-web-config" (OuterVolumeSpecName: "web-config") pod "2213804f-ea04-431e-9cd8-62fb95d6664f" (UID: "2213804f-ea04-431e-9cd8-62fb95d6664f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:56.776829 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776789 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.776829 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776827 2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-config-out\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.776829 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776838 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdpw8\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-kube-api-access-hdpw8\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.777022 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776847 2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-main-db\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.777022 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776856 2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-config-volume\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.777022 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776864 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-main-tls\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.777022 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776872 2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2213804f-ea04-431e-9cd8-62fb95d6664f-tls-assets\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.777022 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776879 2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-web-config\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.777022 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776887 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.777022 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776896 2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-cluster-tls-config\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.777022 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776903 2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2213804f-ea04-431e-9cd8-62fb95d6664f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:56.777022 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:56.776912 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2213804f-ea04-431e-9cd8-62fb95d6664f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:43:57.494967 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.494932 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2213804f-ea04-431e-9cd8-62fb95d6664f","Type":"ContainerDied","Data":"1128875b7c67c062b6e50cc8a16f07408a6d2fc2d6327bbda15123417cd9efe4"} Jul 07 15:43:57.495365 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.495006 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.495365 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.495022 2583 scope.go:117] "RemoveContainer" containerID="188de352fc83e99e271a2a7b8c02fa8efa880346dbc5dd0e2510efe97bd73783" Jul 07 15:43:57.502819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.502800 2583 scope.go:117] "RemoveContainer" containerID="0c8f213ac7420d45a002af90718e0b29e8137cd12413fdaa4dfe9c6d038888b2" Jul 07 15:43:57.509418 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.509404 2583 scope.go:117] "RemoveContainer" containerID="ad2bad1f14938d23a730d4e1d850b69adf15b6aa5403ab4caad3e8a80c1264f6" Jul 07 15:43:57.515555 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.515527 2583 scope.go:117] "RemoveContainer" containerID="a7b870f3a39cfc8278c8a62217cf57b5ba7c953dbe71a13b9553f3cea57520e6" Jul 07 15:43:57.527280 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.520887 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 15:43:57.527280 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.523335 2583 scope.go:117] "RemoveContainer" containerID="1c7ef4d1309f3c9b516b4cfdffa271217a1f94146006754b27fc2243cc74e670" Jul 07 15:43:57.547135 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.547113 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 15:43:57.548213 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.548188 2583 scope.go:117] "RemoveContainer" containerID="2fd200afa30321071c9b30602060e635484602ad39cb7d4d8396dd295cd9009d" Jul 07 15:43:57.555603 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.555576 2583 scope.go:117] "RemoveContainer" containerID="1546bb9acc29b22a9224b20746a34ab91731d5cfce3f17a9de9ace8eb9e82448" Jul 07 15:43:57.563529 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563513 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 15:43:57.563820 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563804 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49273c59-63a8-4691-9077-6abae82484c5" containerName="console" Jul 07 15:43:57.563820 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563819 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="49273c59-63a8-4691-9077-6abae82484c5" containerName="console" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563832 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="alertmanager" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563838 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="alertmanager" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563849 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="init-config-reloader" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563854 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="init-config-reloader" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563861 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy-metric" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563867 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy-metric" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563880 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy-web" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563887 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy-web" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563901 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563909 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563921 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="prom-label-proxy" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563928 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="prom-label-proxy" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563943 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="config-reloader" Jul 07 15:43:57.563952 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.563951 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="config-reloader" Jul 07 15:43:57.564406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.564017 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy-metric" Jul 07 15:43:57.564406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.564030 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy" Jul 07 15:43:57.564406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.564040 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="alertmanager" Jul 07 15:43:57.564406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.564050 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="prom-label-proxy" Jul 07 15:43:57.564406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.564058 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="49273c59-63a8-4691-9077-6abae82484c5" containerName="console" Jul 07 15:43:57.564406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.564066 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="config-reloader" Jul 07 15:43:57.564406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.564075 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" containerName="kube-rbac-proxy-web" Jul 07 15:43:57.567938 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.567922 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.570760 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.570738 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 15:43:57.570841 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.570805 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 15:43:57.570841 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.570739 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cn6k8\"" Jul 07 15:43:57.570942 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.570740 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 15:43:57.570997 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.570981 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 15:43:57.571200 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.571184 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 15:43:57.571534 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.571518 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 15:43:57.571603 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.571588 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 15:43:57.571653 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.571596 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 15:43:57.578498 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.578480 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 15:43:57.591678 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.591657 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 15:43:57.684930 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.684843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f7f655-a2c4-4409-90d8-146d74a5f46c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.684930 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.684887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685133 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.684941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685133 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.684984 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-config-volume\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685133 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.685001 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70f7f655-a2c4-4409-90d8-146d74a5f46c-config-out\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685133 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.685054 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-web-config\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685133 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.685111 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70f7f655-a2c4-4409-90d8-146d74a5f46c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685365 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.685136 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685365 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.685162 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685365 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.685188 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f7f655-a2c4-4409-90d8-146d74a5f46c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685365 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.685216 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8npf\" (UniqueName: \"kubernetes.io/projected/70f7f655-a2c4-4409-90d8-146d74a5f46c-kube-api-access-s8npf\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685365 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.685247 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.685365 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.685270 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70f7f655-a2c4-4409-90d8-146d74a5f46c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.786643 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786606 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-config-volume\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.786643 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70f7f655-a2c4-4409-90d8-146d74a5f46c-config-out\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.786821 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-web-config\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.786821 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786702 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70f7f655-a2c4-4409-90d8-146d74a5f46c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.786821 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786718 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.786821 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786734 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.786821 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f7f655-a2c4-4409-90d8-146d74a5f46c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.786821 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8npf\" (UniqueName: \"kubernetes.io/projected/70f7f655-a2c4-4409-90d8-146d74a5f46c-kube-api-access-s8npf\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.787114 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786836 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.787114 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786867 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70f7f655-a2c4-4409-90d8-146d74a5f46c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.787114 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786906 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f7f655-a2c4-4409-90d8-146d74a5f46c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.787114 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.787114 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.786958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.788018 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.787994 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f7f655-a2c4-4409-90d8-146d74a5f46c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.788406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.788383 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70f7f655-a2c4-4409-90d8-146d74a5f46c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.789890 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.789682 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70f7f655-a2c4-4409-90d8-146d74a5f46c-config-out\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.789890 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.789757 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.789890 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.789768 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.789890 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.789797 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-web-config\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.789890 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.789846 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70f7f655-a2c4-4409-90d8-146d74a5f46c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.790195 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.789977 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.790294 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.790275 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f7f655-a2c4-4409-90d8-146d74a5f46c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.790636 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.790618 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-config-volume\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.791138 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.791116 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.791695 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.791678 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70f7f655-a2c4-4409-90d8-146d74a5f46c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.798433 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.798413 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8npf\" (UniqueName: \"kubernetes.io/projected/70f7f655-a2c4-4409-90d8-146d74a5f46c-kube-api-access-s8npf\") pod \"alertmanager-main-0\" (UID: \"70f7f655-a2c4-4409-90d8-146d74a5f46c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:57.877315 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:57.877291 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 15:43:58.031627 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:58.031605 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 15:43:58.033153 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:43:58.033128 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f7f655_a2c4_4409_90d8_146d74a5f46c.slice/crio-94b6d7f732747e12c2473d1454cba1fefb155e5a89b76ebd9ad614c24d332df5 WatchSource:0}: Error finding container 94b6d7f732747e12c2473d1454cba1fefb155e5a89b76ebd9ad614c24d332df5: Status 404 returned error can't find the container with id 94b6d7f732747e12c2473d1454cba1fefb155e5a89b76ebd9ad614c24d332df5 Jul 07 15:43:58.501364 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:58.501328 2583 generic.go:356] "Generic (PLEG): container finished" podID="70f7f655-a2c4-4409-90d8-146d74a5f46c" containerID="c59376133bd7cc0deefe5da4927aa770c8bfde270e89d8695db52a6372e06301" exitCode=0 Jul 07 15:43:58.501814 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:58.501416 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f7f655-a2c4-4409-90d8-146d74a5f46c","Type":"ContainerDied","Data":"c59376133bd7cc0deefe5da4927aa770c8bfde270e89d8695db52a6372e06301"} Jul 07 15:43:58.501814 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:58.501502 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f7f655-a2c4-4409-90d8-146d74a5f46c","Type":"ContainerStarted","Data":"94b6d7f732747e12c2473d1454cba1fefb155e5a89b76ebd9ad614c24d332df5"} Jul 07 15:43:58.798556 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:58.798518 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2213804f-ea04-431e-9cd8-62fb95d6664f" path="/var/lib/kubelet/pods/2213804f-ea04-431e-9cd8-62fb95d6664f/volumes" Jul 07 15:43:59.436617 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.436570 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f49554c84-w4tmd"] Jul 07 15:43:59.439388 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.439369 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.442712 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.442664 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 15:43:59.442966 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.442926 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 15:43:59.442966 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.442945 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 15:43:59.443125 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.442969 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 15:43:59.443125 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.442992 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n86bz\"" Jul 07 15:43:59.443269 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.443242 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 15:43:59.448657 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.448633 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 15:43:59.455190 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.455172 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f49554c84-w4tmd"] Jul 07 15:43:59.510435 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.510402 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f7f655-a2c4-4409-90d8-146d74a5f46c","Type":"ContainerStarted","Data":"7a9982396092baf8fa0c6cc230775932b4cd935eb33c323559d8acd5f5abb94d"} Jul 07 15:43:59.511074 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.511054 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f7f655-a2c4-4409-90d8-146d74a5f46c","Type":"ContainerStarted","Data":"a6e106596e5898f08f7e6fcea47c1bab85695656b43a011c660dd8c1f3dbc506"} Jul 07 15:43:59.511249 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.511234 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f7f655-a2c4-4409-90d8-146d74a5f46c","Type":"ContainerStarted","Data":"db7c932a9f229b6e49dd04ab2815d663c25587d15c23042dcec6acb59a037e4b"} Jul 07 15:43:59.511376 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.511345 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f7f655-a2c4-4409-90d8-146d74a5f46c","Type":"ContainerStarted","Data":"398dac0ac1b3413534b118ab3beccb02c3153b7b75c7d5b907a10dd65b4c986c"} Jul 07 15:43:59.511490 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.511391 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f7f655-a2c4-4409-90d8-146d74a5f46c","Type":"ContainerStarted","Data":"87b7b9849cb7c6645318fd55825e24449c360fffe4c45d229df3a30b8b3c5cba"} Jul 07 15:43:59.511490 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.511428 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f7f655-a2c4-4409-90d8-146d74a5f46c","Type":"ContainerStarted","Data":"075035f4856a71ec866885093a4e933f8a56ec5fdd006ad9b11a7c5b3363f00d"} Jul 07 15:43:59.551083 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.551031 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.541863793 podStartE2EDuration="2.551014449s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.150656ms" isStatefulPod=false podCreationTimestamp="2026-07-07 15:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:43:59.550310456 +0000 UTC m=+145.680723603" watchObservedRunningTime="2026-07-07 15:43:59.551014449 +0000 UTC m=+145.681427595" Jul 07 15:43:59.603901 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.603863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-telemeter-client-tls\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.603901 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.603901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-federate-client-tls\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.604124 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.604086 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-secret-telemeter-client\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.604189 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.604136 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0721b254-5812-4a3f-878e-cf0b8ec01df5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.604245 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.604200 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0721b254-5812-4a3f-878e-cf0b8ec01df5-serving-certs-ca-bundle\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.604320 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.604295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.604438 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.604414 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwx2l\" (UniqueName: \"kubernetes.io/projected/0721b254-5812-4a3f-878e-cf0b8ec01df5-kube-api-access-xwx2l\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.604514 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.604473 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0721b254-5812-4a3f-878e-cf0b8ec01df5-metrics-client-ca\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.705613 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.705503 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-telemeter-client-tls\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.705613 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.705563 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-federate-client-tls\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.705613 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.705609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-secret-telemeter-client\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.705852 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.705637 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0721b254-5812-4a3f-878e-cf0b8ec01df5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.705852 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.705672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0721b254-5812-4a3f-878e-cf0b8ec01df5-serving-certs-ca-bundle\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.705852 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.705727 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.705852 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.705778 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwx2l\" (UniqueName: \"kubernetes.io/projected/0721b254-5812-4a3f-878e-cf0b8ec01df5-kube-api-access-xwx2l\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.705852 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.705810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0721b254-5812-4a3f-878e-cf0b8ec01df5-metrics-client-ca\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.706510 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.706472 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0721b254-5812-4a3f-878e-cf0b8ec01df5-serving-certs-ca-bundle\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.706662 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.706524 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0721b254-5812-4a3f-878e-cf0b8ec01df5-metrics-client-ca\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.706662 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.706604 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0721b254-5812-4a3f-878e-cf0b8ec01df5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.708160 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.708139 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-secret-telemeter-client\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.708287 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.708264 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-federate-client-tls\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.708404 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.708384 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.708452 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.708394 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0721b254-5812-4a3f-878e-cf0b8ec01df5-telemeter-client-tls\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.716459 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.716416 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwx2l\" (UniqueName: \"kubernetes.io/projected/0721b254-5812-4a3f-878e-cf0b8ec01df5-kube-api-access-xwx2l\") pod \"telemeter-client-7f49554c84-w4tmd\" (UID: \"0721b254-5812-4a3f-878e-cf0b8ec01df5\") " pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.742562 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.742514 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 15:43:59.743433 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.742912 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" gracePeriod=600 Jul 07 15:43:59.743433 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.743294 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy-web" containerID="cri-o://e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" gracePeriod=600 Jul 07 15:43:59.743433 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.743362 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="thanos-sidecar" containerID="cri-o://c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" gracePeriod=600 Jul 07 15:43:59.743722 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.743521 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="config-reloader" containerID="cri-o://813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" gracePeriod=600 Jul 07 15:43:59.743722 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.743642 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy" containerID="cri-o://c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" gracePeriod=600 Jul 07 15:43:59.745220 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.744884 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="prometheus" containerID="cri-o://521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" gracePeriod=600 Jul 07 15:43:59.752074 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.752051 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" Jul 07 15:43:59.914842 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.914813 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f49554c84-w4tmd"] Jul 07 15:43:59.916831 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:43:59.916800 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0721b254_5812_4a3f_878e_cf0b8ec01df5.slice/crio-c1c18b05b11221b214dcb7623d0a693f0e95b7ea2ce5cc80c1422a868f68db64 WatchSource:0}: Error finding container c1c18b05b11221b214dcb7623d0a693f0e95b7ea2ce5cc80c1422a868f68db64: Status 404 returned error can't find the container with id c1c18b05b11221b214dcb7623d0a693f0e95b7ea2ce5cc80c1422a868f68db64 Jul 07 15:43:59.989464 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:43:59.989436 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.109423 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109394 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-tls\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109590 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109449 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109590 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109469 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-web-config\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109590 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109495 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-tls-assets\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109590 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109512 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-metrics-client-certs\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109590 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109531 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-config\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109590 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109586 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109871 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109705 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-serving-certs-ca-bundle\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109871 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109735 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-0\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109871 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109765 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-2\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.109871 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.109808 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht2k9\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-kube-api-access-ht2k9\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.110357 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.110263 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-trusted-ca-bundle\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.110357 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.110318 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-kubelet-serving-ca-bundle\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.110519 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.110367 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-thanos-prometheus-http-client-file\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.110519 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.110410 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-1\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.110519 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.110443 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-db\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.110519 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.110498 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-metrics-client-ca\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.110743 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.110529 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-kube-rbac-proxy\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.110743 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.110587 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-grpc-tls\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.110743 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.110631 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-config-out\") pod \"31afd01a-5b8a-440d-afb1-d1f06be1326b\" (UID: \"31afd01a-5b8a-440d-afb1-d1f06be1326b\") " Jul 07 15:44:00.111639 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.111606 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:00.112314 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.112282 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:00.112961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.112693 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:00.113382 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.113184 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:00.113870 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.113650 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:00.114359 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.114104 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:00.115192 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.115097 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:00.115638 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.115337 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-config" (OuterVolumeSpecName: "config") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:00.116055 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.115854 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116123 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116380 2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116404 2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116421 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116444 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116463 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116480 2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116497 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116519 2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31afd01a-5b8a-440d-afb1-d1f06be1326b-configmap-metrics-client-ca\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116536 2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.116856 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116605 2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-config\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.117370 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116893 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:44:00.117370 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.116903 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-config-out" (OuterVolumeSpecName: "config-out") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:44:00.117984 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.117960 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:00.118125 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.118086 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-kube-api-access-ht2k9" (OuterVolumeSpecName: "kube-api-access-ht2k9") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "kube-api-access-ht2k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:44:00.118241 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.118107 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:44:00.118317 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.118241 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:00.119926 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.119722 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:00.120392 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.120319 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:00.120802 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.120590 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:00.127836 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.127815 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-web-config" (OuterVolumeSpecName: "web-config") pod "31afd01a-5b8a-440d-afb1-d1f06be1326b" (UID: "31afd01a-5b8a-440d-afb1-d1f06be1326b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:00.217663 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217638 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2k9\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-kube-api-access-ht2k9\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.217663 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217661 2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.217819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217673 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-prometheus-k8s-db\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.217819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217681 2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-kube-rbac-proxy\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.217819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217691 2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-grpc-tls\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.217819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217699 2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31afd01a-5b8a-440d-afb1-d1f06be1326b-config-out\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.217819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217708 2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.217819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217716 2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-web-config\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.217819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217724 2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31afd01a-5b8a-440d-afb1-d1f06be1326b-tls-assets\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.217819 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.217732 2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31afd01a-5b8a-440d-afb1-d1f06be1326b-secret-metrics-client-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:00.516466 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.516432 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" event={"ID":"0721b254-5812-4a3f-878e-cf0b8ec01df5","Type":"ContainerStarted","Data":"c1c18b05b11221b214dcb7623d0a693f0e95b7ea2ce5cc80c1422a868f68db64"} Jul 07 15:44:00.519231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519202 2583 generic.go:356] "Generic (PLEG): container finished" podID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerID="4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" exitCode=0 Jul 07 15:44:00.519231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519224 2583 generic.go:356] "Generic (PLEG): container finished" podID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerID="c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" exitCode=0 Jul 07 15:44:00.519231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519230 2583 generic.go:356] "Generic (PLEG): container finished" podID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerID="e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" exitCode=0 Jul 07 15:44:00.519231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519237 2583 generic.go:356] "Generic (PLEG): container finished" podID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerID="c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" exitCode=0 Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519243 2583 generic.go:356] "Generic (PLEG): container finished" podID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerID="813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" exitCode=0 Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519250 2583 generic.go:356] "Generic (PLEG): container finished" podID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerID="521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" exitCode=0 Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519249 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerDied","Data":"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1"} Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519286 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519329 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerDied","Data":"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353"} Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519372 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerDied","Data":"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5"} Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519392 2583 scope.go:117] "RemoveContainer" containerID="4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519415 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerDied","Data":"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3"} Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519458 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerDied","Data":"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8"} Jul 07 15:44:00.519496 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519500 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerDied","Data":"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33"} Jul 07 15:44:00.520001 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.519542 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31afd01a-5b8a-440d-afb1-d1f06be1326b","Type":"ContainerDied","Data":"6ebe3fdf4a2c099c8d9ed62569c5ccb1883cd1c1111dc74786307fdc22a0fe53"} Jul 07 15:44:00.527266 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.527246 2583 scope.go:117] "RemoveContainer" containerID="c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" Jul 07 15:44:00.533928 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.533908 2583 scope.go:117] "RemoveContainer" containerID="e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" Jul 07 15:44:00.540379 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.540362 2583 scope.go:117] "RemoveContainer" containerID="c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" Jul 07 15:44:00.546587 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.546569 2583 scope.go:117] "RemoveContainer" containerID="813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" Jul 07 15:44:00.553144 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.553074 2583 scope.go:117] "RemoveContainer" containerID="521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" Jul 07 15:44:00.558132 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.558109 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 15:44:00.560900 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.560882 2583 scope.go:117] "RemoveContainer" containerID="0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e" Jul 07 15:44:00.560977 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.560944 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 15:44:00.567290 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.567273 2583 scope.go:117] "RemoveContainer" containerID="4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" Jul 07 15:44:00.567511 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:44:00.567494 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": container with ID starting with 4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1 not found: ID does not exist" containerID="4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" Jul 07 15:44:00.567576 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.567519 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1"} err="failed to get container status \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": rpc error: code = NotFound desc = could not find container \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": container with ID starting with 4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1 not found: ID does not exist" Jul 07 15:44:00.567576 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.567536 2583 scope.go:117] "RemoveContainer" containerID="c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" Jul 07 15:44:00.567750 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:44:00.567734 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": container with ID starting with c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353 not found: ID does not exist" containerID="c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" Jul 07 15:44:00.567789 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.567756 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353"} err="failed to get container status \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": rpc error: code = NotFound desc = could not find container \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": container with ID starting with c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353 not found: ID does not exist" Jul 07 15:44:00.567789 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.567771 2583 scope.go:117] "RemoveContainer" containerID="e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" Jul 07 15:44:00.567948 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:44:00.567933 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": container with ID starting with e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5 not found: ID does not exist" containerID="e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" Jul 07 15:44:00.567989 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.567951 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5"} err="failed to get container status \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": rpc error: code = NotFound desc = could not find container \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": container with ID starting with e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5 not found: ID does not exist" Jul 07 15:44:00.567989 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.567964 2583 scope.go:117] "RemoveContainer" containerID="c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" Jul 07 15:44:00.568143 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:44:00.568129 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": container with ID starting with c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3 not found: ID does not exist" containerID="c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" Jul 07 15:44:00.568182 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568146 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3"} err="failed to get container status \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": rpc error: code = NotFound desc = could not find container \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": container with ID starting with c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3 not found: ID does not exist" Jul 07 15:44:00.568182 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568157 2583 scope.go:117] "RemoveContainer" containerID="813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" Jul 07 15:44:00.568299 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:44:00.568284 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": container with ID starting with 813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8 not found: ID does not exist" containerID="813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" Jul 07 15:44:00.568347 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568302 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8"} err="failed to get container status \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": rpc error: code = NotFound desc = could not find container \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": container with ID starting with 813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8 not found: ID does not exist" Jul 07 15:44:00.568347 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568316 2583 scope.go:117] "RemoveContainer" containerID="521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" Jul 07 15:44:00.568494 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:44:00.568480 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": container with ID starting with 521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33 not found: ID does not exist" containerID="521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" Jul 07 15:44:00.568529 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568498 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33"} err="failed to get container status \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": rpc error: code = NotFound desc = could not find container \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": container with ID starting with 521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33 not found: ID does not exist" Jul 07 15:44:00.568529 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568509 2583 scope.go:117] "RemoveContainer" containerID="0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e" Jul 07 15:44:00.568762 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:44:00.568746 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": container with ID starting with 0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e not found: ID does not exist" containerID="0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e" Jul 07 15:44:00.568804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568764 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e"} err="failed to get container status \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": rpc error: code = NotFound desc = could not find container \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": container with ID starting with 0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e not found: ID does not exist" Jul 07 15:44:00.568804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568778 2583 scope.go:117] "RemoveContainer" containerID="4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" Jul 07 15:44:00.568986 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568972 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1"} err="failed to get container status \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": rpc error: code = NotFound desc = could not find container \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": container with ID starting with 4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1 not found: ID does not exist" Jul 07 15:44:00.569028 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.568986 2583 scope.go:117] "RemoveContainer" containerID="c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" Jul 07 15:44:00.569193 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.569175 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353"} err="failed to get container status \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": rpc error: code = NotFound desc = could not find container \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": container with ID starting with c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353 not found: ID does not exist" Jul 07 15:44:00.569236 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.569194 2583 scope.go:117] "RemoveContainer" containerID="e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" Jul 07 15:44:00.569409 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.569391 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5"} err="failed to get container status \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": rpc error: code = NotFound desc = could not find container \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": container with ID starting with e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5 not found: ID does not exist" Jul 07 15:44:00.569451 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.569410 2583 scope.go:117] "RemoveContainer" containerID="c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" Jul 07 15:44:00.569644 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.569629 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3"} err="failed to get container status \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": rpc error: code = NotFound desc = could not find container \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": container with ID starting with c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3 not found: ID does not exist" Jul 07 15:44:00.569697 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.569645 2583 scope.go:117] "RemoveContainer" containerID="813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" Jul 07 15:44:00.569847 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.569834 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8"} err="failed to get container status \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": rpc error: code = NotFound desc = could not find container \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": container with ID starting with 813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8 not found: ID does not exist" Jul 07 15:44:00.569883 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.569847 2583 scope.go:117] "RemoveContainer" containerID="521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" Jul 07 15:44:00.570050 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570034 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33"} err="failed to get container status \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": rpc error: code = NotFound desc = could not find container \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": container with ID starting with 521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33 not found: ID does not exist" Jul 07 15:44:00.570094 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570050 2583 scope.go:117] "RemoveContainer" containerID="0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e" Jul 07 15:44:00.570237 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570222 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e"} err="failed to get container status \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": rpc error: code = NotFound desc = could not find container \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": container with ID starting with 0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e not found: ID does not exist" Jul 07 15:44:00.570275 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570237 2583 scope.go:117] "RemoveContainer" containerID="4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" Jul 07 15:44:00.570415 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570400 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1"} err="failed to get container status \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": rpc error: code = NotFound desc = could not find container \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": container with ID starting with 4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1 not found: ID does not exist" Jul 07 15:44:00.570449 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570415 2583 scope.go:117] "RemoveContainer" containerID="c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" Jul 07 15:44:00.570681 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570662 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353"} err="failed to get container status \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": rpc error: code = NotFound desc = could not find container \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": container with ID starting with c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353 not found: ID does not exist" Jul 07 15:44:00.570725 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570682 2583 scope.go:117] "RemoveContainer" containerID="e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" Jul 07 15:44:00.570877 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570862 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5"} err="failed to get container status \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": rpc error: code = NotFound desc = could not find container \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": container with ID starting with e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5 not found: ID does not exist" Jul 07 15:44:00.570915 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.570878 2583 scope.go:117] "RemoveContainer" containerID="c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" Jul 07 15:44:00.571058 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571026 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3"} err="failed to get container status \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": rpc error: code = NotFound desc = could not find container \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": container with ID starting with c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3 not found: ID does not exist" Jul 07 15:44:00.571097 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571060 2583 scope.go:117] "RemoveContainer" containerID="813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" Jul 07 15:44:00.571231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571216 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8"} err="failed to get container status \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": rpc error: code = NotFound desc = could not find container \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": container with ID starting with 813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8 not found: ID does not exist" Jul 07 15:44:00.571266 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571231 2583 scope.go:117] "RemoveContainer" containerID="521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" Jul 07 15:44:00.571428 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571414 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33"} err="failed to get container status \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": rpc error: code = NotFound desc = could not find container \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": container with ID starting with 521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33 not found: ID does not exist" Jul 07 15:44:00.571474 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571428 2583 scope.go:117] "RemoveContainer" containerID="0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e" Jul 07 15:44:00.571630 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571614 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e"} err="failed to get container status \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": rpc error: code = NotFound desc = could not find container \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": container with ID starting with 0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e not found: ID does not exist" Jul 07 15:44:00.571674 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571637 2583 scope.go:117] "RemoveContainer" containerID="4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" Jul 07 15:44:00.571850 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571834 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1"} err="failed to get container status \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": rpc error: code = NotFound desc = could not find container \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": container with ID starting with 4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1 not found: ID does not exist" Jul 07 15:44:00.571894 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.571851 2583 scope.go:117] "RemoveContainer" containerID="c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" Jul 07 15:44:00.572032 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572014 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353"} err="failed to get container status \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": rpc error: code = NotFound desc = could not find container \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": container with ID starting with c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353 not found: ID does not exist" Jul 07 15:44:00.572074 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572033 2583 scope.go:117] "RemoveContainer" containerID="e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" Jul 07 15:44:00.572210 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572191 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5"} err="failed to get container status \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": rpc error: code = NotFound desc = could not find container \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": container with ID starting with e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5 not found: ID does not exist" Jul 07 15:44:00.572251 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572211 2583 scope.go:117] "RemoveContainer" containerID="c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" Jul 07 15:44:00.572394 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572378 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3"} err="failed to get container status \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": rpc error: code = NotFound desc = could not find container \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": container with ID starting with c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3 not found: ID does not exist" Jul 07 15:44:00.572437 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572394 2583 scope.go:117] "RemoveContainer" containerID="813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" Jul 07 15:44:00.572582 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572567 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8"} err="failed to get container status \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": rpc error: code = NotFound desc = could not find container \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": container with ID starting with 813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8 not found: ID does not exist" Jul 07 15:44:00.572619 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572582 2583 scope.go:117] "RemoveContainer" containerID="521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" Jul 07 15:44:00.572757 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572740 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33"} err="failed to get container status \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": rpc error: code = NotFound desc = could not find container \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": container with ID starting with 521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33 not found: ID does not exist" Jul 07 15:44:00.572799 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572759 2583 scope.go:117] "RemoveContainer" containerID="0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e" Jul 07 15:44:00.572981 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572966 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e"} err="failed to get container status \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": rpc error: code = NotFound desc = could not find container \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": container with ID starting with 0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e not found: ID does not exist" Jul 07 15:44:00.573033 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.572982 2583 scope.go:117] "RemoveContainer" containerID="4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" Jul 07 15:44:00.573180 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.573163 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1"} err="failed to get container status \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": rpc error: code = NotFound desc = could not find container \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": container with ID starting with 4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1 not found: ID does not exist" Jul 07 15:44:00.573218 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.573180 2583 scope.go:117] "RemoveContainer" containerID="c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" Jul 07 15:44:00.573396 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.573377 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353"} err="failed to get container status \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": rpc error: code = NotFound desc = could not find container \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": container with ID starting with c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353 not found: ID does not exist" Jul 07 15:44:00.573435 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.573397 2583 scope.go:117] "RemoveContainer" containerID="e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" Jul 07 15:44:00.573613 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.573598 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5"} err="failed to get container status \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": rpc error: code = NotFound desc = could not find container \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": container with ID starting with e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5 not found: ID does not exist" Jul 07 15:44:00.573658 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.573613 2583 scope.go:117] "RemoveContainer" containerID="c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" Jul 07 15:44:00.573817 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.573802 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3"} err="failed to get container status \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": rpc error: code = NotFound desc = could not find container \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": container with ID starting with c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3 not found: ID does not exist" Jul 07 15:44:00.573884 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.573818 2583 scope.go:117] "RemoveContainer" containerID="813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" Jul 07 15:44:00.574006 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.573991 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8"} err="failed to get container status \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": rpc error: code = NotFound desc = could not find container \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": container with ID starting with 813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8 not found: ID does not exist" Jul 07 15:44:00.574055 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574008 2583 scope.go:117] "RemoveContainer" containerID="521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" Jul 07 15:44:00.574179 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574164 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33"} err="failed to get container status \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": rpc error: code = NotFound desc = could not find container \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": container with ID starting with 521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33 not found: ID does not exist" Jul 07 15:44:00.574222 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574179 2583 scope.go:117] "RemoveContainer" containerID="0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e" Jul 07 15:44:00.574348 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574334 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e"} err="failed to get container status \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": rpc error: code = NotFound desc = could not find container \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": container with ID starting with 0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e not found: ID does not exist" Jul 07 15:44:00.574388 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574349 2583 scope.go:117] "RemoveContainer" containerID="4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1" Jul 07 15:44:00.574522 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574507 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1"} err="failed to get container status \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": rpc error: code = NotFound desc = could not find container \"4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1\": container with ID starting with 4f494d3291d65db3c8251aad5632bfde47bdba78bd9f999cd6aacca9e60a0fb1 not found: ID does not exist" Jul 07 15:44:00.574605 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574522 2583 scope.go:117] "RemoveContainer" containerID="c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353" Jul 07 15:44:00.574719 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574704 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353"} err="failed to get container status \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": rpc error: code = NotFound desc = could not find container \"c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353\": container with ID starting with c0b4e13216ebcd1eae10c34d9e35ffb8a978f4ae96ac108414459840f3ee1353 not found: ID does not exist" Jul 07 15:44:00.574759 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574720 2583 scope.go:117] "RemoveContainer" containerID="e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5" Jul 07 15:44:00.574910 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574890 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5"} err="failed to get container status \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": rpc error: code = NotFound desc = could not find container \"e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5\": container with ID starting with e9dbcf06d7c5ac4ba277b1e0935c64d6b1c2c4b52012109e42f59801d2055da5 not found: ID does not exist" Jul 07 15:44:00.574953 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.574912 2583 scope.go:117] "RemoveContainer" containerID="c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3" Jul 07 15:44:00.575098 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.575082 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3"} err="failed to get container status \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": rpc error: code = NotFound desc = could not find container \"c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3\": container with ID starting with c5aaba8ad95d7f78bbdebe527300f9f3ec667c89e8cf0ce886b9aa5179c904d3 not found: ID does not exist" Jul 07 15:44:00.575139 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.575098 2583 scope.go:117] "RemoveContainer" containerID="813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8" Jul 07 15:44:00.575308 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.575292 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8"} err="failed to get container status \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": rpc error: code = NotFound desc = could not find container \"813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8\": container with ID starting with 813c85aea82fafb8d70286982e74f75f569aae184f4eb3936ae68390f45440e8 not found: ID does not exist" Jul 07 15:44:00.575353 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.575309 2583 scope.go:117] "RemoveContainer" containerID="521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33" Jul 07 15:44:00.575498 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.575478 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33"} err="failed to get container status \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": rpc error: code = NotFound desc = could not find container \"521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33\": container with ID starting with 521b543a9f9309c14b40b26ae27c056719ced598892b8c69de815b741b2eac33 not found: ID does not exist" Jul 07 15:44:00.575536 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.575499 2583 scope.go:117] "RemoveContainer" containerID="0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e" Jul 07 15:44:00.575726 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.575707 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e"} err="failed to get container status \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": rpc error: code = NotFound desc = could not find container \"0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e\": container with ID starting with 0f909e6225829b532e1ce36f99ac7e9a56dedcf0ebf8b7238dc237efd928904e not found: ID does not exist" Jul 07 15:44:00.607261 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607239 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 15:44:00.607702 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607686 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="prometheus" Jul 07 15:44:00.607702 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607703 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="prometheus" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607715 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607724 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607737 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="thanos-sidecar" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607745 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="thanos-sidecar" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607755 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="init-config-reloader" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607766 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="init-config-reloader" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607778 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="config-reloader" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607787 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="config-reloader" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607801 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy-web" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607809 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy-web" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607820 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy-thanos" Jul 07 15:44:00.607853 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607828 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy-thanos" Jul 07 15:44:00.608377 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607967 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy-web" Jul 07 15:44:00.608377 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607980 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy" Jul 07 15:44:00.608377 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.607992 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="config-reloader" Jul 07 15:44:00.608377 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.608002 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="thanos-sidecar" Jul 07 15:44:00.608377 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.608012 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="prometheus" Jul 07 15:44:00.608377 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.608022 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" containerName="kube-rbac-proxy-thanos" Jul 07 15:44:00.611123 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.611099 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.618706 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.618583 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 15:44:00.618706 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.618583 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 15:44:00.619366 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.619307 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 15:44:00.619933 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.619913 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 15:44:00.619933 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.619930 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 15:44:00.620078 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.620046 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-87q8p\"" Jul 07 15:44:00.620133 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.620101 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 15:44:00.620231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.620184 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 15:44:00.620286 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.620256 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 15:44:00.620338 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.620319 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1qdbp9vrrcckc\"" Jul 07 15:44:00.620490 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.620476 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 15:44:00.620621 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.620602 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 15:44:00.620692 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.620615 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 15:44:00.620692 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.620654 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 15:44:00.628816 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.628760 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 15:44:00.629392 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.629375 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 15:44:00.642233 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.642176 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 15:44:00.721814 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.721784 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.721961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.721823 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.721961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.721843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.721961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.721876 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.721961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.721903 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.721961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.721918 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.721961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.721938 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.721961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.721959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.721979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722003 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722060 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722075 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722090 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722107 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722128 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-config\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722208 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722142 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvkp\" (UniqueName: \"kubernetes.io/projected/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-kube-api-access-xfvkp\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722221 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722272 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.722456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.722287 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.797454 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.797423 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31afd01a-5b8a-440d-afb1-d1f06be1326b" path="/var/lib/kubelet/pods/31afd01a-5b8a-440d-afb1-d1f06be1326b/volumes" Jul 07 15:44:00.822927 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.822906 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823159 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823256 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823176 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823256 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823256 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-config\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823412 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823268 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvkp\" (UniqueName: \"kubernetes.io/projected/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-kube-api-access-xfvkp\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823412 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823308 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823412 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823412 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823633 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823749 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823749 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823728 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823749 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823900 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823771 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823900 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823900 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823900 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823867 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.823900 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.824145 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.824145 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823943 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.824145 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823968 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.824145 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.823988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.824341 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.824235 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.824341 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.824245 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.824746 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.824562 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.825651 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.824976 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.826089 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.825823 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.827015 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.827446 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.829134 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.829605 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.829638 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.829781 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.829921 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.830034 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.830062 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830113 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.830090 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830685 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.830352 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.830685 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.830663 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-config\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.834570 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.834526 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvkp\" (UniqueName: \"kubernetes.io/projected/3a68f7a5-a2df-41a0-8658-ab4cd6b91f40-kube-api-access-xfvkp\") pod \"prometheus-k8s-0\" (UID: \"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:00.922747 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:00.922669 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:01.072229 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:01.072196 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 15:44:01.075318 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:44:01.075284 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a68f7a5_a2df_41a0_8658_ab4cd6b91f40.slice/crio-5b4335d7f4c2d3673b23d13e9f8efd40646e6a4bc0c2bf9c960fd054738f295d WatchSource:0}: Error finding container 5b4335d7f4c2d3673b23d13e9f8efd40646e6a4bc0c2bf9c960fd054738f295d: Status 404 returned error can't find the container with id 5b4335d7f4c2d3673b23d13e9f8efd40646e6a4bc0c2bf9c960fd054738f295d Jul 07 15:44:01.528057 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:01.528017 2583 generic.go:356] "Generic (PLEG): container finished" podID="3a68f7a5-a2df-41a0-8658-ab4cd6b91f40" containerID="e751a96bb2b96fc9509751292afcc166e58ab5b5d227173a3359103693f6c6d2" exitCode=0 Jul 07 15:44:01.528481 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:01.528095 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40","Type":"ContainerDied","Data":"e751a96bb2b96fc9509751292afcc166e58ab5b5d227173a3359103693f6c6d2"} Jul 07 15:44:01.528481 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:01.528219 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40","Type":"ContainerStarted","Data":"5b4335d7f4c2d3673b23d13e9f8efd40646e6a4bc0c2bf9c960fd054738f295d"} Jul 07 15:44:02.538578 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.538475 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40","Type":"ContainerStarted","Data":"2c73190dca17bfb2f70ca96f06c87d799cf4ab3c3b5f7a2663b0e286c7cea7b8"} Jul 07 15:44:02.538578 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.538570 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40","Type":"ContainerStarted","Data":"9354474340c3843aa9c55d6ef5995cd89ecf34ee0c79b540c62b8dc3bd4b5879"} Jul 07 15:44:02.538981 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.538617 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40","Type":"ContainerStarted","Data":"e91c5058a8b838358c2188c8fc7ec849f3dc065907a46e7e4ffb7adb70719220"} Jul 07 15:44:02.538981 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.538657 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40","Type":"ContainerStarted","Data":"39553bc7e8634be6c95a08272a80fa010eed1dc3897778917b2546752f0e8a8f"} Jul 07 15:44:02.538981 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.538693 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40","Type":"ContainerStarted","Data":"16df72c60369e3a0293ce58cd9f1bad4a144f87164605a5da9c06beafcbbd4e6"} Jul 07 15:44:02.538981 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.538734 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a68f7a5-a2df-41a0-8658-ab4cd6b91f40","Type":"ContainerStarted","Data":"952ba283c21b4ecb68a260eb72fdefb48fab5d135eaf15208f184daceb9cbc5d"} Jul 07 15:44:02.540585 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.540537 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" event={"ID":"0721b254-5812-4a3f-878e-cf0b8ec01df5","Type":"ContainerStarted","Data":"6636839bb814a576a7646b5c719b8fdad61483922ba22c992be147793f0e3a30"} Jul 07 15:44:02.540691 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.540612 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" event={"ID":"0721b254-5812-4a3f-878e-cf0b8ec01df5","Type":"ContainerStarted","Data":"acbedd96e73a519ff47c319d1652d38d7396f8d30c4d99bde3344dea855a6955"} Jul 07 15:44:02.540691 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.540639 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" event={"ID":"0721b254-5812-4a3f-878e-cf0b8ec01df5","Type":"ContainerStarted","Data":"b8c7082500414a8a34acfd3412abbe4df964a1b30448b8a22a7495daf86b5e83"} Jul 07 15:44:02.578710 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.578660 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.560925869 podStartE2EDuration="2.578645835s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.719966ms" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:02.576618736 +0000 UTC m=+148.707031883" watchObservedRunningTime="2026-07-07 15:44:02.578645835 +0000 UTC m=+148.709058980" Jul 07 15:44:02.605262 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:02.605216 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f49554c84-w4tmd" podStartSLOduration=1.5962618979999998 podStartE2EDuration="3.605203602s" totalImagesPullingTime="2.008941704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:02.603921765 +0000 UTC m=+148.734334948" watchObservedRunningTime="2026-07-07 15:44:02.605203602 +0000 UTC m=+148.735616747" Jul 07 15:44:05.923204 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:05.923172 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:44:14.359217 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:14.359187 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4d74674-qvm6x"] Jul 07 15:44:39.378958 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.378896 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6c4d74674-qvm6x" podUID="5243c61f-74fe-49d0-859e-780879e50208" containerName="console" containerID="cri-o://1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430" gracePeriod=15 Jul 07 15:44:39.613964 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.613942 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:44:39.693229 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.693141 2583 generic.go:356] "Generic (PLEG): container finished" podID="5243c61f-74fe-49d0-859e-780879e50208" containerID="1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430" exitCode=0 Jul 07 15:44:39.693229 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.693187 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4d74674-qvm6x" Jul 07 15:44:39.693229 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.693209 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4d74674-qvm6x" event={"ID":"5243c61f-74fe-49d0-859e-780879e50208","Type":"ContainerDied","Data":"1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430"} Jul 07 15:44:39.693437 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.693250 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4d74674-qvm6x" event={"ID":"5243c61f-74fe-49d0-859e-780879e50208","Type":"ContainerDied","Data":"80404fbc77f31458b1c258bab99740fa5d856313e16f644ed0a2f6e4f07c5a65"} Jul 07 15:44:39.693437 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.693268 2583 scope.go:117] "RemoveContainer" containerID="1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430" Jul 07 15:44:39.701226 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.701211 2583 scope.go:117] "RemoveContainer" containerID="1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430" Jul 07 15:44:39.701477 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:44:39.701460 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430\": container with ID starting with 1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430 not found: ID does not exist" containerID="1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430" Jul 07 15:44:39.701568 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.701485 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430"} err="failed to get container status \"1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430\": rpc error: code = NotFound desc = could not find container \"1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430\": container with ID starting with 1bc582a5283fb1d58205d8670ede355b65c9efff4f9eaa122e9cd89073d58430 not found: ID does not exist" Jul 07 15:44:39.764946 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.764901 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp4ld\" (UniqueName: \"kubernetes.io/projected/5243c61f-74fe-49d0-859e-780879e50208-kube-api-access-qp4ld\") pod \"5243c61f-74fe-49d0-859e-780879e50208\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " Jul 07 15:44:39.765052 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.764959 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-trusted-ca-bundle\") pod \"5243c61f-74fe-49d0-859e-780879e50208\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " Jul 07 15:44:39.765052 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.764984 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-oauth-config\") pod \"5243c61f-74fe-49d0-859e-780879e50208\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " Jul 07 15:44:39.765052 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.765004 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-oauth-serving-cert\") pod \"5243c61f-74fe-49d0-859e-780879e50208\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " Jul 07 15:44:39.765169 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.765051 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-serving-cert\") pod \"5243c61f-74fe-49d0-859e-780879e50208\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " Jul 07 15:44:39.765169 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.765109 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-service-ca\") pod \"5243c61f-74fe-49d0-859e-780879e50208\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " Jul 07 15:44:39.765169 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.765137 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-console-config\") pod \"5243c61f-74fe-49d0-859e-780879e50208\" (UID: \"5243c61f-74fe-49d0-859e-780879e50208\") " Jul 07 15:44:39.765436 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.765407 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5243c61f-74fe-49d0-859e-780879e50208" (UID: "5243c61f-74fe-49d0-859e-780879e50208"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:39.765596 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.765531 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5243c61f-74fe-49d0-859e-780879e50208" (UID: "5243c61f-74fe-49d0-859e-780879e50208"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:39.765697 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.765606 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-service-ca" (OuterVolumeSpecName: "service-ca") pod "5243c61f-74fe-49d0-859e-780879e50208" (UID: "5243c61f-74fe-49d0-859e-780879e50208"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:39.765697 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.765651 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-console-config" (OuterVolumeSpecName: "console-config") pod "5243c61f-74fe-49d0-859e-780879e50208" (UID: "5243c61f-74fe-49d0-859e-780879e50208"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:44:39.767319 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.767289 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5243c61f-74fe-49d0-859e-780879e50208-kube-api-access-qp4ld" (OuterVolumeSpecName: "kube-api-access-qp4ld") pod "5243c61f-74fe-49d0-859e-780879e50208" (UID: "5243c61f-74fe-49d0-859e-780879e50208"). InnerVolumeSpecName "kube-api-access-qp4ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:44:39.767420 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.767300 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5243c61f-74fe-49d0-859e-780879e50208" (UID: "5243c61f-74fe-49d0-859e-780879e50208"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:39.767420 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.767346 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5243c61f-74fe-49d0-859e-780879e50208" (UID: "5243c61f-74fe-49d0-859e-780879e50208"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:44:39.866628 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.866580 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-service-ca\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:39.866628 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.866620 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-console-config\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:39.866628 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.866630 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp4ld\" (UniqueName: \"kubernetes.io/projected/5243c61f-74fe-49d0-859e-780879e50208-kube-api-access-qp4ld\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:39.866628 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.866641 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-trusted-ca-bundle\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:39.866892 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.866649 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-oauth-config\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:39.866892 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.866658 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5243c61f-74fe-49d0-859e-780879e50208-oauth-serving-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:39.866892 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:39.866668 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5243c61f-74fe-49d0-859e-780879e50208-console-serving-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:44:40.020340 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:40.020314 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4d74674-qvm6x"] Jul 07 15:44:40.025302 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:40.025278 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4d74674-qvm6x"] Jul 07 15:44:40.796772 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:44:40.796739 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5243c61f-74fe-49d0-859e-780879e50208" path="/var/lib/kubelet/pods/5243c61f-74fe-49d0-859e-780879e50208/volumes" Jul 07 15:45:00.923370 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:45:00.923333 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:45:00.938573 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:45:00.938526 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:45:01.792511 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:45:01.792486 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:46:31.687463 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.687436 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-grbzd"] Jul 07 15:46:31.687931 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.687787 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5243c61f-74fe-49d0-859e-780879e50208" containerName="console" Jul 07 15:46:31.687931 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.687799 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5243c61f-74fe-49d0-859e-780879e50208" containerName="console" Jul 07 15:46:31.687931 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.687850 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5243c61f-74fe-49d0-859e-780879e50208" containerName="console" Jul 07 15:46:31.690720 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.690706 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:31.693406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.693387 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 15:46:31.693534 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.693443 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 15:46:31.695863 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.695847 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6q2zq\"" Jul 07 15:46:31.700727 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.700706 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-grbzd"] Jul 07 15:46:31.801685 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.801658 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-grbzd\" (UID: \"a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351\") " pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:31.801832 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.801710 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72rlh\" (UniqueName: \"kubernetes.io/projected/a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351-kube-api-access-72rlh\") pod \"cert-manager-webhook-6486ffd997-grbzd\" (UID: \"a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351\") " pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:31.902435 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.902393 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72rlh\" (UniqueName: \"kubernetes.io/projected/a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351-kube-api-access-72rlh\") pod \"cert-manager-webhook-6486ffd997-grbzd\" (UID: \"a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351\") " pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:31.902632 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.902480 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-grbzd\" (UID: \"a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351\") " pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:31.912790 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.912765 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-grbzd\" (UID: \"a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351\") " pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:31.912978 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:31.912956 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72rlh\" (UniqueName: \"kubernetes.io/projected/a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351-kube-api-access-72rlh\") pod \"cert-manager-webhook-6486ffd997-grbzd\" (UID: \"a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351\") " pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:32.007360 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:32.007280 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:32.155887 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:32.155859 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-grbzd"] Jul 07 15:46:32.164079 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:46:32.164050 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4e2ef96_f43f_4ee8_a78a_33b6fd6d5351.slice/crio-533e46629540359c156ccee7bd54b5a021a4fe6115a563fb346bc2d6e08014af WatchSource:0}: Error finding container 533e46629540359c156ccee7bd54b5a021a4fe6115a563fb346bc2d6e08014af: Status 404 returned error can't find the container with id 533e46629540359c156ccee7bd54b5a021a4fe6115a563fb346bc2d6e08014af Jul 07 15:46:33.129074 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:33.129032 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" event={"ID":"a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351","Type":"ContainerStarted","Data":"533e46629540359c156ccee7bd54b5a021a4fe6115a563fb346bc2d6e08014af"} Jul 07 15:46:35.025767 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:35.025738 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:46:35.028064 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:35.026875 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:46:35.028469 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:35.028456 2583 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 15:46:35.138804 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:35.138767 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" event={"ID":"a4e2ef96-f43f-4ee8-a78a-33b6fd6d5351","Type":"ContainerStarted","Data":"b754db2f1dc9124995996773d2f04cff92c6ead1f66db52919f2bb0b1d46995b"} Jul 07 15:46:35.138976 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:35.138884 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:35.183059 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:35.182870 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" podStartSLOduration=1.2851792770000001 podStartE2EDuration="4.182850056s" totalImagesPullingTime="2.897670779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:46:35.182334502 +0000 UTC m=+301.312747650" watchObservedRunningTime="2026-07-07 15:46:35.182850056 +0000 UTC m=+301.313263202" Jul 07 15:46:41.144888 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.144855 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-grbzd" Jul 07 15:46:41.416085 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.416003 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7bbkj"] Jul 07 15:46:41.419239 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.419224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7bbkj" Jul 07 15:46:41.422144 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.422126 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5mqwq\"" Jul 07 15:46:41.431457 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.431436 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7bbkj"] Jul 07 15:46:41.491876 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.491843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrbkb\" (UniqueName: \"kubernetes.io/projected/f8d49889-295a-45eb-ad50-992a8c753647-kube-api-access-jrbkb\") pod \"cert-manager-55b4d954d5-7bbkj\" (UID: \"f8d49889-295a-45eb-ad50-992a8c753647\") " pod="cert-manager/cert-manager-55b4d954d5-7bbkj" Jul 07 15:46:41.492057 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.491946 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d49889-295a-45eb-ad50-992a8c753647-bound-sa-token\") pod \"cert-manager-55b4d954d5-7bbkj\" (UID: \"f8d49889-295a-45eb-ad50-992a8c753647\") " pod="cert-manager/cert-manager-55b4d954d5-7bbkj" Jul 07 15:46:41.592846 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.592806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrbkb\" (UniqueName: \"kubernetes.io/projected/f8d49889-295a-45eb-ad50-992a8c753647-kube-api-access-jrbkb\") pod \"cert-manager-55b4d954d5-7bbkj\" (UID: \"f8d49889-295a-45eb-ad50-992a8c753647\") " pod="cert-manager/cert-manager-55b4d954d5-7bbkj" Jul 07 15:46:41.593051 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.592888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d49889-295a-45eb-ad50-992a8c753647-bound-sa-token\") pod \"cert-manager-55b4d954d5-7bbkj\" (UID: \"f8d49889-295a-45eb-ad50-992a8c753647\") " pod="cert-manager/cert-manager-55b4d954d5-7bbkj" Jul 07 15:46:41.602645 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.602616 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8d49889-295a-45eb-ad50-992a8c753647-bound-sa-token\") pod \"cert-manager-55b4d954d5-7bbkj\" (UID: \"f8d49889-295a-45eb-ad50-992a8c753647\") " pod="cert-manager/cert-manager-55b4d954d5-7bbkj" Jul 07 15:46:41.602767 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.602750 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrbkb\" (UniqueName: \"kubernetes.io/projected/f8d49889-295a-45eb-ad50-992a8c753647-kube-api-access-jrbkb\") pod \"cert-manager-55b4d954d5-7bbkj\" (UID: \"f8d49889-295a-45eb-ad50-992a8c753647\") " pod="cert-manager/cert-manager-55b4d954d5-7bbkj" Jul 07 15:46:41.727886 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.727782 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7bbkj" Jul 07 15:46:41.853775 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:41.853751 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7bbkj"] Jul 07 15:46:41.856249 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:46:41.856225 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8d49889_295a_45eb_ad50_992a8c753647.slice/crio-e4364bda3b8f73302f3eb6219da7d0f76dd45c11fe9e15984209dd73c075f28d WatchSource:0}: Error finding container e4364bda3b8f73302f3eb6219da7d0f76dd45c11fe9e15984209dd73c075f28d: Status 404 returned error can't find the container with id e4364bda3b8f73302f3eb6219da7d0f76dd45c11fe9e15984209dd73c075f28d Jul 07 15:46:42.167313 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:42.167282 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7bbkj" event={"ID":"f8d49889-295a-45eb-ad50-992a8c753647","Type":"ContainerStarted","Data":"0ef71d5c130586a1d98dcc2c799bd1d91d53a9d071fa7e4958c81ed9c6863a57"} Jul 07 15:46:42.167708 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:42.167322 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7bbkj" event={"ID":"f8d49889-295a-45eb-ad50-992a8c753647","Type":"ContainerStarted","Data":"e4364bda3b8f73302f3eb6219da7d0f76dd45c11fe9e15984209dd73c075f28d"} Jul 07 15:46:42.188260 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:46:42.188215 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7bbkj" podStartSLOduration=1.188200089 podStartE2EDuration="1.188200089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:46:42.187774983 +0000 UTC m=+308.318188129" watchObservedRunningTime="2026-07-07 15:46:42.188200089 +0000 UTC m=+308.318613234" Jul 07 15:47:05.053385 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.053345 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7444699877-kqrqp"] Jul 07 15:47:05.056984 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.056966 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.060931 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.060910 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 15:47:05.061046 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.060958 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 15:47:05.062305 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.062288 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:47:05.062406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.062366 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 15:47:05.062406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.062367 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 15:47:05.062535 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.062499 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-df7tn\"" Jul 07 15:47:05.077201 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.077177 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7444699877-kqrqp"] Jul 07 15:47:05.192052 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.192020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/417000df-04e6-4ca8-af71-6a678f95e4cb-metrics-cert\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.192052 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.192057 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/417000df-04e6-4ca8-af71-6a678f95e4cb-manager-config\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.192274 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.192076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/417000df-04e6-4ca8-af71-6a678f95e4cb-cert\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.192274 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.192100 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvspf\" (UniqueName: \"kubernetes.io/projected/417000df-04e6-4ca8-af71-6a678f95e4cb-kube-api-access-kvspf\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.293386 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.293351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/417000df-04e6-4ca8-af71-6a678f95e4cb-manager-config\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.293386 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.293390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/417000df-04e6-4ca8-af71-6a678f95e4cb-cert\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.293652 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.293416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvspf\" (UniqueName: \"kubernetes.io/projected/417000df-04e6-4ca8-af71-6a678f95e4cb-kube-api-access-kvspf\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.293652 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.293528 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/417000df-04e6-4ca8-af71-6a678f95e4cb-metrics-cert\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.294104 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.294079 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/417000df-04e6-4ca8-af71-6a678f95e4cb-manager-config\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.295950 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.295929 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/417000df-04e6-4ca8-af71-6a678f95e4cb-cert\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.296214 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.296192 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/417000df-04e6-4ca8-af71-6a678f95e4cb-metrics-cert\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.304211 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.304153 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvspf\" (UniqueName: \"kubernetes.io/projected/417000df-04e6-4ca8-af71-6a678f95e4cb-kube-api-access-kvspf\") pod \"lws-controller-manager-7444699877-kqrqp\" (UID: \"417000df-04e6-4ca8-af71-6a678f95e4cb\") " pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.366926 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.366896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:05.502837 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.502810 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7444699877-kqrqp"] Jul 07 15:47:05.504168 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:47:05.504141 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417000df_04e6_4ca8_af71_6a678f95e4cb.slice/crio-4b8c1ed3501fb7b5d2918a7cd794f18a352f83c1b2f2662debbe8a5b60f2e337 WatchSource:0}: Error finding container 4b8c1ed3501fb7b5d2918a7cd794f18a352f83c1b2f2662debbe8a5b60f2e337: Status 404 returned error can't find the container with id 4b8c1ed3501fb7b5d2918a7cd794f18a352f83c1b2f2662debbe8a5b60f2e337 Jul 07 15:47:05.506071 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.506056 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:47:05.594926 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.594896 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85"] Jul 07 15:47:05.599442 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.599423 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.603430 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.603414 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 15:47:05.604151 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.604134 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kvnwl\"" Jul 07 15:47:05.604251 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.604160 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 15:47:05.604377 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.604360 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 15:47:05.605071 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.605054 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 15:47:05.670635 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.670608 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85"] Jul 07 15:47:05.697004 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.696976 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c3503fe-bc75-4ace-a40c-af18bac782e9-webhook-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.697151 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.697010 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c3503fe-bc75-4ace-a40c-af18bac782e9-tmp\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.697151 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.697039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c3503fe-bc75-4ace-a40c-af18bac782e9-manifests\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.697151 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.697064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c3503fe-bc75-4ace-a40c-af18bac782e9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.697151 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.697108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7sf\" (UniqueName: \"kubernetes.io/projected/3c3503fe-bc75-4ace-a40c-af18bac782e9-kube-api-access-dh7sf\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.797968 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.797936 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c3503fe-bc75-4ace-a40c-af18bac782e9-webhook-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.798146 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.797978 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c3503fe-bc75-4ace-a40c-af18bac782e9-tmp\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.798146 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.798104 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c3503fe-bc75-4ace-a40c-af18bac782e9-manifests\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.798146 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.798141 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c3503fe-bc75-4ace-a40c-af18bac782e9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.798307 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.798166 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7sf\" (UniqueName: \"kubernetes.io/projected/3c3503fe-bc75-4ace-a40c-af18bac782e9-kube-api-access-dh7sf\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.798456 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.798428 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c3503fe-bc75-4ace-a40c-af18bac782e9-manifests\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.800187 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.800167 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c3503fe-bc75-4ace-a40c-af18bac782e9-tmp\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.800398 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.800383 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c3503fe-bc75-4ace-a40c-af18bac782e9-webhook-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.800446 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.800430 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c3503fe-bc75-4ace-a40c-af18bac782e9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.808006 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.807984 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7sf\" (UniqueName: \"kubernetes.io/projected/3c3503fe-bc75-4ace-a40c-af18bac782e9-kube-api-access-dh7sf\") pod \"opendatahub-operator-controller-manager-685dff6fcf-ghv85\" (UID: \"3c3503fe-bc75-4ace-a40c-af18bac782e9\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:05.910098 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:05.910004 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:06.056222 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:06.056190 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85"] Jul 07 15:47:06.059330 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:47:06.059302 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3503fe_bc75_4ace_a40c_af18bac782e9.slice/crio-8837bc7cdd287621b4f7b672d7046ed7b01e24951001d33564ce29653693db5d WatchSource:0}: Error finding container 8837bc7cdd287621b4f7b672d7046ed7b01e24951001d33564ce29653693db5d: Status 404 returned error can't find the container with id 8837bc7cdd287621b4f7b672d7046ed7b01e24951001d33564ce29653693db5d Jul 07 15:47:06.267931 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:06.267816 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" event={"ID":"417000df-04e6-4ca8-af71-6a678f95e4cb","Type":"ContainerStarted","Data":"4b8c1ed3501fb7b5d2918a7cd794f18a352f83c1b2f2662debbe8a5b60f2e337"} Jul 07 15:47:06.269818 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:06.269785 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" event={"ID":"3c3503fe-bc75-4ace-a40c-af18bac782e9","Type":"ContainerStarted","Data":"8837bc7cdd287621b4f7b672d7046ed7b01e24951001d33564ce29653693db5d"} Jul 07 15:47:10.292432 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:10.292399 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" event={"ID":"417000df-04e6-4ca8-af71-6a678f95e4cb","Type":"ContainerStarted","Data":"8dede0e7bab455874d4fcd79aade3ae86cdcfb70a6563ae8bfccdb047dab5eae"} Jul 07 15:47:10.292883 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:10.292455 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:10.293894 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:10.293871 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" event={"ID":"3c3503fe-bc75-4ace-a40c-af18bac782e9","Type":"ContainerStarted","Data":"25f51d3c9aa61b41203ac374911e570c6cc916163cd46fdbf99e34acb96c8b81"} Jul 07 15:47:10.315503 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:10.315460 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" podStartSLOduration=1.193218105 podStartE2EDuration="5.315447171s" totalImagesPullingTime="4.122229066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:10.314863565 +0000 UTC m=+336.445276714" watchObservedRunningTime="2026-07-07 15:47:10.315447171 +0000 UTC m=+336.445860317" Jul 07 15:47:11.299992 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:11.299957 2583 generic.go:356] "Generic (PLEG): container finished" podID="3c3503fe-bc75-4ace-a40c-af18bac782e9" containerID="25f51d3c9aa61b41203ac374911e570c6cc916163cd46fdbf99e34acb96c8b81" exitCode=0 Jul 07 15:47:11.300400 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:11.300040 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" event={"ID":"3c3503fe-bc75-4ace-a40c-af18bac782e9","Type":"ContainerDied","Data":"25f51d3c9aa61b41203ac374911e570c6cc916163cd46fdbf99e34acb96c8b81"} Jul 07 15:47:12.306305 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:12.306272 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" event={"ID":"3c3503fe-bc75-4ace-a40c-af18bac782e9","Type":"ContainerStarted","Data":"ce19ba0c4fb5f13f0eec28cf8f425428c35b05dfa2769659c2093d3bc4e4781c"} Jul 07 15:47:12.306728 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:12.306354 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:12.340226 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:12.340179 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" podStartSLOduration=2.367153967 podStartE2EDuration="7.340163002s" totalImagesPullingTime="3.932020583s" totalInitContainerRuntime="1.040988452s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:12.338344545 +0000 UTC m=+338.468757704" watchObservedRunningTime="2026-07-07 15:47:12.340163002 +0000 UTC m=+338.470576148" Jul 07 15:47:21.303200 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:21.303169 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7444699877-kqrqp" Jul 07 15:47:23.315507 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:23.315481 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-ghv85" Jul 07 15:47:26.829883 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.829806 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-ccc59c597-hdv58"] Jul 07 15:47:26.833189 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.833175 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:26.840586 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.840563 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 15:47:26.840699 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.840632 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 15:47:26.841393 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.841377 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 15:47:26.847480 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.847460 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6tbg6\"" Jul 07 15:47:26.847573 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.847532 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 15:47:26.869411 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.869388 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-ccc59c597-hdv58"] Jul 07 15:47:26.886108 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.886085 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f958698-a875-466d-af7f-3a4e6c6528eb-tls-certs\") pod \"kube-auth-proxy-ccc59c597-hdv58\" (UID: \"1f958698-a875-466d-af7f-3a4e6c6528eb\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:26.886221 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.886118 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f958698-a875-466d-af7f-3a4e6c6528eb-tmp\") pod \"kube-auth-proxy-ccc59c597-hdv58\" (UID: \"1f958698-a875-466d-af7f-3a4e6c6528eb\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:26.886221 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.886151 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8zq\" (UniqueName: \"kubernetes.io/projected/1f958698-a875-466d-af7f-3a4e6c6528eb-kube-api-access-jp8zq\") pod \"kube-auth-proxy-ccc59c597-hdv58\" (UID: \"1f958698-a875-466d-af7f-3a4e6c6528eb\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:26.987109 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.987079 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f958698-a875-466d-af7f-3a4e6c6528eb-tls-certs\") pod \"kube-auth-proxy-ccc59c597-hdv58\" (UID: \"1f958698-a875-466d-af7f-3a4e6c6528eb\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:26.987244 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.987114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f958698-a875-466d-af7f-3a4e6c6528eb-tmp\") pod \"kube-auth-proxy-ccc59c597-hdv58\" (UID: \"1f958698-a875-466d-af7f-3a4e6c6528eb\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:26.987244 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.987137 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8zq\" (UniqueName: \"kubernetes.io/projected/1f958698-a875-466d-af7f-3a4e6c6528eb-kube-api-access-jp8zq\") pod \"kube-auth-proxy-ccc59c597-hdv58\" (UID: \"1f958698-a875-466d-af7f-3a4e6c6528eb\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:26.989336 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.989311 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f958698-a875-466d-af7f-3a4e6c6528eb-tmp\") pod \"kube-auth-proxy-ccc59c597-hdv58\" (UID: \"1f958698-a875-466d-af7f-3a4e6c6528eb\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:26.989570 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:26.989532 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f958698-a875-466d-af7f-3a4e6c6528eb-tls-certs\") pod \"kube-auth-proxy-ccc59c597-hdv58\" (UID: \"1f958698-a875-466d-af7f-3a4e6c6528eb\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:27.044231 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:27.044206 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8zq\" (UniqueName: \"kubernetes.io/projected/1f958698-a875-466d-af7f-3a4e6c6528eb-kube-api-access-jp8zq\") pod \"kube-auth-proxy-ccc59c597-hdv58\" (UID: \"1f958698-a875-466d-af7f-3a4e6c6528eb\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:27.142486 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:27.142402 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" Jul 07 15:47:27.408208 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:47:27.408181 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f958698_a875_466d_af7f_3a4e6c6528eb.slice/crio-1065895ba2dcb5429349127108a16f6aca104f16f0a2227e1c6d4fac2a60d660 WatchSource:0}: Error finding container 1065895ba2dcb5429349127108a16f6aca104f16f0a2227e1c6d4fac2a60d660: Status 404 returned error can't find the container with id 1065895ba2dcb5429349127108a16f6aca104f16f0a2227e1c6d4fac2a60d660 Jul 07 15:47:27.411943 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:27.411923 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-ccc59c597-hdv58"] Jul 07 15:47:28.380836 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:28.380805 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" event={"ID":"1f958698-a875-466d-af7f-3a4e6c6528eb","Type":"ContainerStarted","Data":"1065895ba2dcb5429349127108a16f6aca104f16f0a2227e1c6d4fac2a60d660"} Jul 07 15:47:30.391567 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:47:30.391512 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" event={"ID":"1f958698-a875-466d-af7f-3a4e6c6528eb","Type":"ContainerStarted","Data":"2cacb39602ea94bc418ca9236dbd7bfff599aef284036b850003e75b692cfb44"} Jul 07 15:49:06.522238 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.522137 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-ccc59c597-hdv58" podStartSLOduration=98.513126579 podStartE2EDuration="1m40.522118814s" totalImagesPullingTime="2.008992235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:30.425731789 +0000 UTC m=+356.556144941" watchObservedRunningTime="2026-07-07 15:49:06.522118814 +0000 UTC m=+452.652532002" Jul 07 15:49:06.522755 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.522703 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58f4b5668f-brlc2"] Jul 07 15:49:06.526058 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.526039 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.528846 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.528826 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2ljcb\"" Jul 07 15:49:06.528958 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.528857 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 15:49:06.530691 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.530674 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 15:49:06.531043 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.531024 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 15:49:06.531143 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.531080 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 15:49:06.531199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.531153 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 15:49:06.531199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.531172 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 15:49:06.531199 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.531189 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 15:49:06.535652 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.535634 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 15:49:06.538023 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.538005 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f4b5668f-brlc2"] Jul 07 15:49:06.647921 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.647870 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-service-ca\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.648142 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.647990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91298c97-a9ef-4278-858c-5bf93c0a9694-console-oauth-config\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.648142 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.648036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-console-config\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.648142 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.648077 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-oauth-serving-cert\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.648142 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.648099 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpvv\" (UniqueName: \"kubernetes.io/projected/91298c97-a9ef-4278-858c-5bf93c0a9694-kube-api-access-2lpvv\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.648331 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.648152 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91298c97-a9ef-4278-858c-5bf93c0a9694-console-serving-cert\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.648331 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.648216 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-trusted-ca-bundle\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.748927 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.748893 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91298c97-a9ef-4278-858c-5bf93c0a9694-console-oauth-config\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.748927 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.748929 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-console-config\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.749155 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.748954 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-oauth-serving-cert\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.749155 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.748969 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpvv\" (UniqueName: \"kubernetes.io/projected/91298c97-a9ef-4278-858c-5bf93c0a9694-kube-api-access-2lpvv\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.749155 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.749004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91298c97-a9ef-4278-858c-5bf93c0a9694-console-serving-cert\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.749155 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.749041 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-trusted-ca-bundle\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.749155 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.749081 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-service-ca\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.749787 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.749763 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-console-config\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.749918 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.749888 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-service-ca\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.749984 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.749915 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-oauth-serving-cert\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.750200 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.750172 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91298c97-a9ef-4278-858c-5bf93c0a9694-trusted-ca-bundle\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.751575 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.751522 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91298c97-a9ef-4278-858c-5bf93c0a9694-console-serving-cert\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.751575 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.751535 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91298c97-a9ef-4278-858c-5bf93c0a9694-console-oauth-config\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.762665 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.762634 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpvv\" (UniqueName: \"kubernetes.io/projected/91298c97-a9ef-4278-858c-5bf93c0a9694-kube-api-access-2lpvv\") pod \"console-58f4b5668f-brlc2\" (UID: \"91298c97-a9ef-4278-858c-5bf93c0a9694\") " pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.835969 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.835938 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:06.964935 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:06.964912 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f4b5668f-brlc2"] Jul 07 15:49:06.967010 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:49:06.966983 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91298c97_a9ef_4278_858c_5bf93c0a9694.slice/crio-6396ee357b3744655d7a433fdb7075f44d66b6ecee0e2c126168ec34a75d815b WatchSource:0}: Error finding container 6396ee357b3744655d7a433fdb7075f44d66b6ecee0e2c126168ec34a75d815b: Status 404 returned error can't find the container with id 6396ee357b3744655d7a433fdb7075f44d66b6ecee0e2c126168ec34a75d815b Jul 07 15:49:07.829273 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:07.829238 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f4b5668f-brlc2" event={"ID":"91298c97-a9ef-4278-858c-5bf93c0a9694","Type":"ContainerStarted","Data":"f6a09ee8928af0cd5f6923c48f4be021806627312823f0c306bf6f9839059368"} Jul 07 15:49:07.829640 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:07.829287 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f4b5668f-brlc2" event={"ID":"91298c97-a9ef-4278-858c-5bf93c0a9694","Type":"ContainerStarted","Data":"6396ee357b3744655d7a433fdb7075f44d66b6ecee0e2c126168ec34a75d815b"} Jul 07 15:49:07.858435 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:07.858384 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58f4b5668f-brlc2" podStartSLOduration=1.8583705190000002 podStartE2EDuration="1.858370519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:07.857072173 +0000 UTC m=+453.987485353" watchObservedRunningTime="2026-07-07 15:49:07.858370519 +0000 UTC m=+453.988783664" Jul 07 15:49:08.020961 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.020933 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw"] Jul 07 15:49:08.024357 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.024335 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.027470 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.027449 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jglv8\"" Jul 07 15:49:08.027599 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.027485 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 15:49:08.027599 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.027591 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 15:49:08.027701 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.027683 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 15:49:08.027986 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.027931 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 15:49:08.037467 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.037446 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw"] Jul 07 15:49:08.161962 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.161870 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk82k\" (UniqueName: \"kubernetes.io/projected/031f4fa5-906a-40dc-baee-6d1b84859fb7-kube-api-access-nk82k\") pod \"kuadrant-console-plugin-66b5774985-rjdrw\" (UID: \"031f4fa5-906a-40dc-baee-6d1b84859fb7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.161962 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.161909 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/031f4fa5-906a-40dc-baee-6d1b84859fb7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rjdrw\" (UID: \"031f4fa5-906a-40dc-baee-6d1b84859fb7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.162173 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.162025 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/031f4fa5-906a-40dc-baee-6d1b84859fb7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rjdrw\" (UID: \"031f4fa5-906a-40dc-baee-6d1b84859fb7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.262811 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.262780 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/031f4fa5-906a-40dc-baee-6d1b84859fb7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rjdrw\" (UID: \"031f4fa5-906a-40dc-baee-6d1b84859fb7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.262990 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.262846 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/031f4fa5-906a-40dc-baee-6d1b84859fb7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rjdrw\" (UID: \"031f4fa5-906a-40dc-baee-6d1b84859fb7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.262990 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.262898 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk82k\" (UniqueName: \"kubernetes.io/projected/031f4fa5-906a-40dc-baee-6d1b84859fb7-kube-api-access-nk82k\") pod \"kuadrant-console-plugin-66b5774985-rjdrw\" (UID: \"031f4fa5-906a-40dc-baee-6d1b84859fb7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.263519 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.263495 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/031f4fa5-906a-40dc-baee-6d1b84859fb7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rjdrw\" (UID: \"031f4fa5-906a-40dc-baee-6d1b84859fb7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.265210 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.265188 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/031f4fa5-906a-40dc-baee-6d1b84859fb7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rjdrw\" (UID: \"031f4fa5-906a-40dc-baee-6d1b84859fb7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.273279 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.273259 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk82k\" (UniqueName: \"kubernetes.io/projected/031f4fa5-906a-40dc-baee-6d1b84859fb7-kube-api-access-nk82k\") pod \"kuadrant-console-plugin-66b5774985-rjdrw\" (UID: \"031f4fa5-906a-40dc-baee-6d1b84859fb7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.334397 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.334371 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" Jul 07 15:49:08.464131 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.464079 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw"] Jul 07 15:49:08.466138 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:49:08.466114 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod031f4fa5_906a_40dc_baee_6d1b84859fb7.slice/crio-62691efb3fa3191f05d8c0829e643ebbcb7f109900478024633062e6e88bad7a WatchSource:0}: Error finding container 62691efb3fa3191f05d8c0829e643ebbcb7f109900478024633062e6e88bad7a: Status 404 returned error can't find the container with id 62691efb3fa3191f05d8c0829e643ebbcb7f109900478024633062e6e88bad7a Jul 07 15:49:08.834530 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:08.834497 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" event={"ID":"031f4fa5-906a-40dc-baee-6d1b84859fb7","Type":"ContainerStarted","Data":"62691efb3fa3191f05d8c0829e643ebbcb7f109900478024633062e6e88bad7a"} Jul 07 15:49:16.836399 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:16.836358 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:16.836963 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:16.836437 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:16.842370 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:16.842344 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:16.883108 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:16.883081 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58f4b5668f-brlc2" Jul 07 15:49:19.673792 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.673755 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9"] Jul 07 15:49:19.712318 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.712284 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9"] Jul 07 15:49:19.712486 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.712438 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:49:19.715514 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.715494 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dxbl6\"" Jul 07 15:49:19.879960 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.879918 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2946fb9b-5f00-406c-8fea-64f77b830d90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zclp9\" (UID: \"2946fb9b-5f00-406c-8fea-64f77b830d90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:49:19.880141 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.880016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxb6\" (UniqueName: \"kubernetes.io/projected/2946fb9b-5f00-406c-8fea-64f77b830d90-kube-api-access-hjxb6\") pod \"kuadrant-operator-controller-manager-86878c769f-zclp9\" (UID: \"2946fb9b-5f00-406c-8fea-64f77b830d90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:49:19.980980 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.980889 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2946fb9b-5f00-406c-8fea-64f77b830d90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zclp9\" (UID: \"2946fb9b-5f00-406c-8fea-64f77b830d90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:49:19.980980 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.980972 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxb6\" (UniqueName: \"kubernetes.io/projected/2946fb9b-5f00-406c-8fea-64f77b830d90-kube-api-access-hjxb6\") pod \"kuadrant-operator-controller-manager-86878c769f-zclp9\" (UID: \"2946fb9b-5f00-406c-8fea-64f77b830d90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:49:19.981353 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.981328 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2946fb9b-5f00-406c-8fea-64f77b830d90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zclp9\" (UID: \"2946fb9b-5f00-406c-8fea-64f77b830d90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:49:19.992017 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:19.991981 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxb6\" (UniqueName: \"kubernetes.io/projected/2946fb9b-5f00-406c-8fea-64f77b830d90-kube-api-access-hjxb6\") pod \"kuadrant-operator-controller-manager-86878c769f-zclp9\" (UID: \"2946fb9b-5f00-406c-8fea-64f77b830d90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:49:20.023825 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:20.023793 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:49:31.057449 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:31.057421 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9"] Jul 07 15:49:31.059394 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:49:31.059368 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2946fb9b_5f00_406c_8fea_64f77b830d90.slice/crio-593ec9914e4ac18b735d68d72dc890d48cc9bc2c80cc9aa537dff80249da986d WatchSource:0}: Error finding container 593ec9914e4ac18b735d68d72dc890d48cc9bc2c80cc9aa537dff80249da986d: Status 404 returned error can't find the container with id 593ec9914e4ac18b735d68d72dc890d48cc9bc2c80cc9aa537dff80249da986d Jul 07 15:49:31.957057 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:31.957016 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" event={"ID":"031f4fa5-906a-40dc-baee-6d1b84859fb7","Type":"ContainerStarted","Data":"241924fd14f98ec20945c541e919d02309d235aea502b24fcfc1486880c0b9bf"} Jul 07 15:49:31.958325 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:31.958302 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" event={"ID":"2946fb9b-5f00-406c-8fea-64f77b830d90","Type":"ContainerStarted","Data":"593ec9914e4ac18b735d68d72dc890d48cc9bc2c80cc9aa537dff80249da986d"} Jul 07 15:49:31.982625 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:31.982580 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rjdrw" podStartSLOduration=1.149349249 podStartE2EDuration="23.982565838s" totalImagesPullingTime="22.833216589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:31.980353369 +0000 UTC m=+478.110766515" watchObservedRunningTime="2026-07-07 15:49:31.982565838 +0000 UTC m=+478.112978975" Jul 07 15:49:36.984847 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:36.984811 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" event={"ID":"2946fb9b-5f00-406c-8fea-64f77b830d90","Type":"ContainerStarted","Data":"e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b"} Jul 07 15:49:36.985276 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:36.984906 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:49:37.009251 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:37.009200 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" podStartSLOduration=12.274588237 podStartE2EDuration="18.009185434s" totalImagesPullingTime="5.734597197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:37.008087895 +0000 UTC m=+483.138501042" watchObservedRunningTime="2026-07-07 15:49:37.009185434 +0000 UTC m=+483.139598579" Jul 07 15:49:47.992450 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:49:47.992416 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 15:50:08.798077 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:08.798049 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-s6zdp"] Jul 07 15:50:08.863296 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:08.863256 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-s6zdp"] Jul 07 15:50:08.863458 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:08.863350 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-s6zdp" Jul 07 15:50:08.871596 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:08.871574 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ctdxm\"" Jul 07 15:50:09.030609 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:09.030577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6s8\" (UniqueName: \"kubernetes.io/projected/9bdc57ba-ac39-4688-a554-d8166d89b379-kube-api-access-xs6s8\") pod \"authorino-75d556bd77-s6zdp\" (UID: \"9bdc57ba-ac39-4688-a554-d8166d89b379\") " pod="kuadrant-system/authorino-75d556bd77-s6zdp" Jul 07 15:50:09.131512 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:09.131474 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6s8\" (UniqueName: \"kubernetes.io/projected/9bdc57ba-ac39-4688-a554-d8166d89b379-kube-api-access-xs6s8\") pod \"authorino-75d556bd77-s6zdp\" (UID: \"9bdc57ba-ac39-4688-a554-d8166d89b379\") " pod="kuadrant-system/authorino-75d556bd77-s6zdp" Jul 07 15:50:09.150889 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:09.150860 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6s8\" (UniqueName: \"kubernetes.io/projected/9bdc57ba-ac39-4688-a554-d8166d89b379-kube-api-access-xs6s8\") pod \"authorino-75d556bd77-s6zdp\" (UID: \"9bdc57ba-ac39-4688-a554-d8166d89b379\") " pod="kuadrant-system/authorino-75d556bd77-s6zdp" Jul 07 15:50:09.172461 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:09.172437 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-s6zdp" Jul 07 15:50:09.307672 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:09.307649 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-s6zdp"] Jul 07 15:50:09.308869 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:50:09.308840 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bdc57ba_ac39_4688_a554_d8166d89b379.slice/crio-46066d94db322d0a97d8288fe657e3ef45f56e94e46f7ae42b9a505160c6cac6 WatchSource:0}: Error finding container 46066d94db322d0a97d8288fe657e3ef45f56e94e46f7ae42b9a505160c6cac6: Status 404 returned error can't find the container with id 46066d94db322d0a97d8288fe657e3ef45f56e94e46f7ae42b9a505160c6cac6 Jul 07 15:50:10.142325 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:10.142275 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-s6zdp" event={"ID":"9bdc57ba-ac39-4688-a554-d8166d89b379","Type":"ContainerStarted","Data":"46066d94db322d0a97d8288fe657e3ef45f56e94e46f7ae42b9a505160c6cac6"} Jul 07 15:50:13.160854 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:13.160812 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-s6zdp" event={"ID":"9bdc57ba-ac39-4688-a554-d8166d89b379","Type":"ContainerStarted","Data":"2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05"} Jul 07 15:50:13.194630 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:13.194585 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-s6zdp" podStartSLOduration=2.048446102 podStartE2EDuration="5.194568922s" totalImagesPullingTime="3.14612282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:13.193190136 +0000 UTC m=+519.323603283" watchObservedRunningTime="2026-07-07 15:50:13.194568922 +0000 UTC m=+519.324982068" Jul 07 15:50:43.234119 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:43.234076 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-s6zdp"] Jul 07 15:50:43.234705 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:43.234288 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-s6zdp" podUID="9bdc57ba-ac39-4688-a554-d8166d89b379" containerName="authorino" containerID="cri-o://2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05" gracePeriod=30 Jul 07 15:50:43.494008 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:43.493958 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-s6zdp" Jul 07 15:50:43.641843 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:43.641816 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs6s8\" (UniqueName: \"kubernetes.io/projected/9bdc57ba-ac39-4688-a554-d8166d89b379-kube-api-access-xs6s8\") pod \"9bdc57ba-ac39-4688-a554-d8166d89b379\" (UID: \"9bdc57ba-ac39-4688-a554-d8166d89b379\") " Jul 07 15:50:43.643967 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:43.643942 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdc57ba-ac39-4688-a554-d8166d89b379-kube-api-access-xs6s8" (OuterVolumeSpecName: "kube-api-access-xs6s8") pod "9bdc57ba-ac39-4688-a554-d8166d89b379" (UID: "9bdc57ba-ac39-4688-a554-d8166d89b379"). InnerVolumeSpecName "kube-api-access-xs6s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:50:43.743254 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:43.743225 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6s8\" (UniqueName: \"kubernetes.io/projected/9bdc57ba-ac39-4688-a554-d8166d89b379-kube-api-access-xs6s8\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 15:50:44.305969 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.305934 2583 generic.go:356] "Generic (PLEG): container finished" podID="9bdc57ba-ac39-4688-a554-d8166d89b379" containerID="2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05" exitCode=0 Jul 07 15:50:44.306450 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.305981 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-s6zdp" Jul 07 15:50:44.306450 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.306025 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-s6zdp" event={"ID":"9bdc57ba-ac39-4688-a554-d8166d89b379","Type":"ContainerDied","Data":"2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05"} Jul 07 15:50:44.306450 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.306068 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-s6zdp" event={"ID":"9bdc57ba-ac39-4688-a554-d8166d89b379","Type":"ContainerDied","Data":"46066d94db322d0a97d8288fe657e3ef45f56e94e46f7ae42b9a505160c6cac6"} Jul 07 15:50:44.306450 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.306087 2583 scope.go:117] "RemoveContainer" containerID="2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05" Jul 07 15:50:44.314353 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.314339 2583 scope.go:117] "RemoveContainer" containerID="2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05" Jul 07 15:50:44.314624 ip-10-0-128-121 kubenswrapper[2583]: E0707 15:50:44.314603 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05\": container with ID starting with 2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05 not found: ID does not exist" containerID="2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05" Jul 07 15:50:44.314725 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.314630 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05"} err="failed to get container status \"2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05\": rpc error: code = NotFound desc = could not find container \"2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05\": container with ID starting with 2410654610d994f771330bc4862b6901694bc18d0094a379e6f3417f2be92a05 not found: ID does not exist" Jul 07 15:50:44.338673 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.338653 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-s6zdp"] Jul 07 15:50:44.340173 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.340151 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-s6zdp"] Jul 07 15:50:44.797275 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:50:44.797243 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bdc57ba-ac39-4688-a554-d8166d89b379" path="/var/lib/kubelet/pods/9bdc57ba-ac39-4688-a554-d8166d89b379/volumes" Jul 07 15:51:24.897178 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.897141 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-r7ldp"] Jul 07 15:51:24.897687 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.897510 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bdc57ba-ac39-4688-a554-d8166d89b379" containerName="authorino" Jul 07 15:51:24.897687 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.897521 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bdc57ba-ac39-4688-a554-d8166d89b379" containerName="authorino" Jul 07 15:51:24.897687 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.897615 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bdc57ba-ac39-4688-a554-d8166d89b379" containerName="authorino" Jul 07 15:51:24.900995 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.900976 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:51:24.904977 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.904955 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 07 15:51:24.908603 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.908585 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-ctm2s"] Jul 07 15:51:24.912104 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.912088 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:24.919522 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.919495 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-r7ldp"] Jul 07 15:51:24.922839 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:24.922810 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-ctm2s"] Jul 07 15:51:25.033638 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.033599 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp5rx\" (UniqueName: \"kubernetes.io/projected/b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a-kube-api-access-tp5rx\") pod \"payload-pre-processing-7964457696-r7ldp\" (UID: \"b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a\") " pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:51:25.033638 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.033638 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc19cca-d26f-4954-ba6b-c524423efd2f-plugins-config-volume\") pod \"payload-processing-58f86c7798-ctm2s\" (UID: \"cbc19cca-d26f-4954-ba6b-c524423efd2f\") " pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:25.033859 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.033718 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmf4t\" (UniqueName: \"kubernetes.io/projected/cbc19cca-d26f-4954-ba6b-c524423efd2f-kube-api-access-lmf4t\") pod \"payload-processing-58f86c7798-ctm2s\" (UID: \"cbc19cca-d26f-4954-ba6b-c524423efd2f\") " pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:25.033859 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.033746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-r7ldp\" (UID: \"b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a\") " pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:51:25.134707 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.134665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp5rx\" (UniqueName: \"kubernetes.io/projected/b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a-kube-api-access-tp5rx\") pod \"payload-pre-processing-7964457696-r7ldp\" (UID: \"b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a\") " pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:51:25.134707 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.134704 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc19cca-d26f-4954-ba6b-c524423efd2f-plugins-config-volume\") pod \"payload-processing-58f86c7798-ctm2s\" (UID: \"cbc19cca-d26f-4954-ba6b-c524423efd2f\") " pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:25.134926 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.134771 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmf4t\" (UniqueName: \"kubernetes.io/projected/cbc19cca-d26f-4954-ba6b-c524423efd2f-kube-api-access-lmf4t\") pod \"payload-processing-58f86c7798-ctm2s\" (UID: \"cbc19cca-d26f-4954-ba6b-c524423efd2f\") " pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:25.134926 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.134802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-r7ldp\" (UID: \"b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a\") " pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:51:25.135729 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.135699 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-r7ldp\" (UID: \"b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a\") " pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:51:25.135846 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.135700 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc19cca-d26f-4954-ba6b-c524423efd2f-plugins-config-volume\") pod \"payload-processing-58f86c7798-ctm2s\" (UID: \"cbc19cca-d26f-4954-ba6b-c524423efd2f\") " pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:25.146120 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.146085 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp5rx\" (UniqueName: \"kubernetes.io/projected/b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a-kube-api-access-tp5rx\") pod \"payload-pre-processing-7964457696-r7ldp\" (UID: \"b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a\") " pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:51:25.146681 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.146663 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmf4t\" (UniqueName: \"kubernetes.io/projected/cbc19cca-d26f-4954-ba6b-c524423efd2f-kube-api-access-lmf4t\") pod \"payload-processing-58f86c7798-ctm2s\" (UID: \"cbc19cca-d26f-4954-ba6b-c524423efd2f\") " pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:25.211709 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.211612 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:51:25.223895 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.223865 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:25.365340 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.365310 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-r7ldp"] Jul 07 15:51:25.366148 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:51:25.366108 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3302ffd_4765_4a5a_9ee7_87a7f4b0a47a.slice/crio-6be2cb6b0355b32b4234f4619ec25cbdbe837d4af9c2c3b2ae0557397ef97754 WatchSource:0}: Error finding container 6be2cb6b0355b32b4234f4619ec25cbdbe837d4af9c2c3b2ae0557397ef97754: Status 404 returned error can't find the container with id 6be2cb6b0355b32b4234f4619ec25cbdbe837d4af9c2c3b2ae0557397ef97754 Jul 07 15:51:25.392204 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.392178 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-ctm2s"] Jul 07 15:51:25.393755 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:51:25.393730 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc19cca_d26f_4954_ba6b_c524423efd2f.slice/crio-8beb0bb71d5118a4f58ce009d8badf926f37e0c25114b996160f302de3c8381a WatchSource:0}: Error finding container 8beb0bb71d5118a4f58ce009d8badf926f37e0c25114b996160f302de3c8381a: Status 404 returned error can't find the container with id 8beb0bb71d5118a4f58ce009d8badf926f37e0c25114b996160f302de3c8381a Jul 07 15:51:25.501656 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.501566 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" event={"ID":"b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a","Type":"ContainerStarted","Data":"6be2cb6b0355b32b4234f4619ec25cbdbe837d4af9c2c3b2ae0557397ef97754"} Jul 07 15:51:25.502603 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:25.502580 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" event={"ID":"cbc19cca-d26f-4954-ba6b-c524423efd2f","Type":"ContainerStarted","Data":"8beb0bb71d5118a4f58ce009d8badf926f37e0c25114b996160f302de3c8381a"} Jul 07 15:51:27.517868 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:27.517822 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" event={"ID":"cbc19cca-d26f-4954-ba6b-c524423efd2f","Type":"ContainerStarted","Data":"164cd80775334f6685dded8a1b529f5757d98e7a114143ae70c4d5cd3956268f"} Jul 07 15:51:27.518334 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:27.517950 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:27.519347 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:27.519326 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" event={"ID":"b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a","Type":"ContainerStarted","Data":"ebd2ff9c45bbacdfd1c20eb08e993fe9f5a5209589f5a886e5f54906b5965e0e"} Jul 07 15:51:27.519485 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:27.519467 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:51:27.539534 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:27.539484 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" podStartSLOduration=2.137830886 podStartE2EDuration="3.539469368s" totalImagesPullingTime="1.401638482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:27.538232786 +0000 UTC m=+593.668645933" watchObservedRunningTime="2026-07-07 15:51:27.539469368 +0000 UTC m=+593.669882569" Jul 07 15:51:27.558574 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:27.558497 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" podStartSLOduration=2.143519435 podStartE2EDuration="3.558480515s" totalImagesPullingTime="1.41496108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:27.557846145 +0000 UTC m=+593.688259292" watchObservedRunningTime="2026-07-07 15:51:27.558480515 +0000 UTC m=+593.688893668" Jul 07 15:51:35.058071 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:35.057962 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:51:35.063515 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:35.058436 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:51:38.526093 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:38.526065 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-ctm2s" Jul 07 15:51:38.526484 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:51:38.526156 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-r7ldp" Jul 07 15:52:15.324891 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.324858 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52"] Jul 07 15:52:15.328589 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.328566 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.331362 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.331345 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 15:52:15.332572 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.332534 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4zfhc\"" Jul 07 15:52:15.332678 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.332601 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 07 15:52:15.332678 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.332614 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 15:52:15.340932 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.340909 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52"] Jul 07 15:52:15.500492 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.500458 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.500690 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.500505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.500690 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.500533 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.500690 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.500649 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f555cd03-9179-4f54-af32-b4c00d95be92-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.500798 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.500687 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.500798 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.500746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvrg\" (UniqueName: \"kubernetes.io/projected/f555cd03-9179-4f54-af32-b4c00d95be92-kube-api-access-vwvrg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.602067 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.601974 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvrg\" (UniqueName: \"kubernetes.io/projected/f555cd03-9179-4f54-af32-b4c00d95be92-kube-api-access-vwvrg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.602067 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.602037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.602287 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.602091 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.602287 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.602131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.602287 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.602191 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f555cd03-9179-4f54-af32-b4c00d95be92-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.602287 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.602227 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.602553 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.602519 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.602693 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.602537 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.602833 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.602813 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.604517 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.604497 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f555cd03-9179-4f54-af32-b4c00d95be92-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.604703 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.604686 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f555cd03-9179-4f54-af32-b4c00d95be92-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.613043 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.613019 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvrg\" (UniqueName: \"kubernetes.io/projected/f555cd03-9179-4f54-af32-b4c00d95be92-kube-api-access-vwvrg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bkl52\" (UID: \"f555cd03-9179-4f54-af32-b4c00d95be92\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.639863 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.639831 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:15.779619 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.779593 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52"] Jul 07 15:52:15.781529 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:52:15.781501 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf555cd03_9179_4f54_af32_b4c00d95be92.slice/crio-efb0acd11bed3334e12b06b30ab2174274e2581cdc94267ca4e6a74f7107e654 WatchSource:0}: Error finding container efb0acd11bed3334e12b06b30ab2174274e2581cdc94267ca4e6a74f7107e654: Status 404 returned error can't find the container with id efb0acd11bed3334e12b06b30ab2174274e2581cdc94267ca4e6a74f7107e654 Jul 07 15:52:15.783241 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:15.783218 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:52:16.753875 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:16.753836 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" event={"ID":"f555cd03-9179-4f54-af32-b4c00d95be92","Type":"ContainerStarted","Data":"efb0acd11bed3334e12b06b30ab2174274e2581cdc94267ca4e6a74f7107e654"} Jul 07 15:52:17.760013 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:17.759975 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" event={"ID":"f555cd03-9179-4f54-af32-b4c00d95be92","Type":"ContainerStarted","Data":"911b6e5e6f2e1a3ddb50667036fac4f59069fc6a62ed718592668487b47c16aa"} Jul 07 15:52:17.780205 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:17.780158 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" podStartSLOduration=1.4792936 podStartE2EDuration="2.780142483s" totalImagesPullingTime="1.300848883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:17.779358362 +0000 UTC m=+643.909771500" watchObservedRunningTime="2026-07-07 15:52:17.780142483 +0000 UTC m=+643.910555628" Jul 07 15:52:25.640536 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:25.640491 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:25.641029 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:25.640587 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:25.653363 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:25.653338 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:52:25.818438 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:52:25.818406 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bkl52" Jul 07 15:53:10.756202 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.756170 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x"] Jul 07 15:53:10.759959 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.759939 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.762781 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.762759 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 15:53:10.774921 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.774899 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x"] Jul 07 15:53:10.782614 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.782593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.782723 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.782635 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.782723 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.782669 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.782723 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.782716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.782858 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.782741 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rzb\" (UniqueName: \"kubernetes.io/projected/486b7d60-f27f-48df-a78a-63afc74dca44-kube-api-access-h6rzb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.782858 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.782838 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486b7d60-f27f-48df-a78a-63afc74dca44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.884218 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.884189 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.884218 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.884225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.884406 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.884248 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.884462 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.884440 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.884517 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.884483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rzb\" (UniqueName: \"kubernetes.io/projected/486b7d60-f27f-48df-a78a-63afc74dca44-kube-api-access-h6rzb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.884586 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.884566 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486b7d60-f27f-48df-a78a-63afc74dca44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.884644 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.884624 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.884700 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.884674 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.884822 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.884799 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.886463 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.886442 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486b7d60-f27f-48df-a78a-63afc74dca44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.886886 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.886871 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486b7d60-f27f-48df-a78a-63afc74dca44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:10.895499 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:10.895479 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rzb\" (UniqueName: \"kubernetes.io/projected/486b7d60-f27f-48df-a78a-63afc74dca44-kube-api-access-h6rzb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x\" (UID: \"486b7d60-f27f-48df-a78a-63afc74dca44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:11.069417 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:11.069381 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:11.210062 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:11.210031 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x"] Jul 07 15:53:11.211622 ip-10-0-128-121 kubenswrapper[2583]: W0707 15:53:11.211598 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod486b7d60_f27f_48df_a78a_63afc74dca44.slice/crio-1c0d454a15556fde8e6a220f445f5d4684ced096c2c4ee02bb993225a583b36d WatchSource:0}: Error finding container 1c0d454a15556fde8e6a220f445f5d4684ced096c2c4ee02bb993225a583b36d: Status 404 returned error can't find the container with id 1c0d454a15556fde8e6a220f445f5d4684ced096c2c4ee02bb993225a583b36d Jul 07 15:53:12.014577 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:12.014522 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" event={"ID":"486b7d60-f27f-48df-a78a-63afc74dca44","Type":"ContainerStarted","Data":"28cd162406a63dcd362bf8f1f2de0f81d23d8205f3899dba08798e3655d9998b"} Jul 07 15:53:12.014927 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:12.014587 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" event={"ID":"486b7d60-f27f-48df-a78a-63afc74dca44","Type":"ContainerStarted","Data":"1c0d454a15556fde8e6a220f445f5d4684ced096c2c4ee02bb993225a583b36d"} Jul 07 15:53:21.070263 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:21.070229 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:21.070768 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:21.070290 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:21.084188 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:21.084157 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:53:21.110388 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:21.110334 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" podStartSLOduration=10.882544591 podStartE2EDuration="11.110318519s" totalImagesPullingTime="227.773928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:53:12.046473662 +0000 UTC m=+698.176886805" watchObservedRunningTime="2026-07-07 15:53:21.110318519 +0000 UTC m=+707.240731664" Jul 07 15:53:22.074126 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:53:22.074097 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x" Jul 07 15:56:35.089306 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:56:35.089280 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 15:56:35.091355 ip-10-0-128-121 kubenswrapper[2583]: I0707 15:56:35.089727 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:00:19.488287 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:19.488248 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9"] Jul 07 16:00:19.488840 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:19.488452 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" podUID="2946fb9b-5f00-406c-8fea-64f77b830d90" containerName="manager" containerID="cri-o://e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b" gracePeriod=10 Jul 07 16:00:22.237189 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:22.237165 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 16:00:22.312756 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:22.312720 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjxb6\" (UniqueName: \"kubernetes.io/projected/2946fb9b-5f00-406c-8fea-64f77b830d90-kube-api-access-hjxb6\") pod \"2946fb9b-5f00-406c-8fea-64f77b830d90\" (UID: \"2946fb9b-5f00-406c-8fea-64f77b830d90\") " Jul 07 16:00:22.312932 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:22.312791 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2946fb9b-5f00-406c-8fea-64f77b830d90-extensions-socket-volume\") pod \"2946fb9b-5f00-406c-8fea-64f77b830d90\" (UID: \"2946fb9b-5f00-406c-8fea-64f77b830d90\") " Jul 07 16:00:22.313193 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:22.313170 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2946fb9b-5f00-406c-8fea-64f77b830d90-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2946fb9b-5f00-406c-8fea-64f77b830d90" (UID: "2946fb9b-5f00-406c-8fea-64f77b830d90"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:00:22.314760 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:22.314734 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2946fb9b-5f00-406c-8fea-64f77b830d90-kube-api-access-hjxb6" (OuterVolumeSpecName: "kube-api-access-hjxb6") pod "2946fb9b-5f00-406c-8fea-64f77b830d90" (UID: "2946fb9b-5f00-406c-8fea-64f77b830d90"). InnerVolumeSpecName "kube-api-access-hjxb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:00:22.414348 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:22.414260 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjxb6\" (UniqueName: \"kubernetes.io/projected/2946fb9b-5f00-406c-8fea-64f77b830d90-kube-api-access-hjxb6\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:00:22.414348 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:22.414291 2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2946fb9b-5f00-406c-8fea-64f77b830d90-extensions-socket-volume\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:00:23.109429 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:23.109396 2583 generic.go:356] "Generic (PLEG): container finished" podID="2946fb9b-5f00-406c-8fea-64f77b830d90" containerID="e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b" exitCode=0 Jul 07 16:00:23.109612 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:23.109442 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" Jul 07 16:00:23.109612 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:23.109459 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" event={"ID":"2946fb9b-5f00-406c-8fea-64f77b830d90","Type":"ContainerDied","Data":"e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b"} Jul 07 16:00:23.109612 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:23.109506 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9" event={"ID":"2946fb9b-5f00-406c-8fea-64f77b830d90","Type":"ContainerDied","Data":"593ec9914e4ac18b735d68d72dc890d48cc9bc2c80cc9aa537dff80249da986d"} Jul 07 16:00:23.109612 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:23.109530 2583 scope.go:117] "RemoveContainer" containerID="e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b" Jul 07 16:00:23.119251 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:23.119231 2583 scope.go:117] "RemoveContainer" containerID="e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b" Jul 07 16:00:23.119511 ip-10-0-128-121 kubenswrapper[2583]: E0707 16:00:23.119489 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b\": container with ID starting with e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b not found: ID does not exist" containerID="e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b" Jul 07 16:00:23.119568 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:23.119520 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b"} err="failed to get container status \"e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b\": rpc error: code = NotFound desc = could not find container \"e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b\": container with ID starting with e289104d5083173da04f364c03267dd7b8ae96c882e17fa2f8b3f7db2f9bf33b not found: ID does not exist" Jul 07 16:00:23.134782 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:23.134747 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9"] Jul 07 16:00:23.141363 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:23.141338 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zclp9"] Jul 07 16:00:24.798342 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:00:24.798311 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2946fb9b-5f00-406c-8fea-64f77b830d90" path="/var/lib/kubelet/pods/2946fb9b-5f00-406c-8fea-64f77b830d90/volumes" Jul 07 16:01:20.629851 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.629815 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk"] Jul 07 16:01:20.630306 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.630190 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2946fb9b-5f00-406c-8fea-64f77b830d90" containerName="manager" Jul 07 16:01:20.630306 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.630201 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2946fb9b-5f00-406c-8fea-64f77b830d90" containerName="manager" Jul 07 16:01:20.630306 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.630286 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2946fb9b-5f00-406c-8fea-64f77b830d90" containerName="manager" Jul 07 16:01:20.633364 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.633347 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:20.636333 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.636308 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dxbl6\"" Jul 07 16:01:20.647264 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.647239 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk"] Jul 07 16:01:20.663842 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.663814 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bxpq\" (UniqueName: \"kubernetes.io/projected/96c645a0-f9e7-4bc9-893f-e3d2218ba133-kube-api-access-5bxpq\") pod \"kuadrant-operator-controller-manager-86878c769f-58lkk\" (UID: \"96c645a0-f9e7-4bc9-893f-e3d2218ba133\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:20.663965 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.663867 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96c645a0-f9e7-4bc9-893f-e3d2218ba133-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-58lkk\" (UID: \"96c645a0-f9e7-4bc9-893f-e3d2218ba133\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:20.764974 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.764942 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96c645a0-f9e7-4bc9-893f-e3d2218ba133-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-58lkk\" (UID: \"96c645a0-f9e7-4bc9-893f-e3d2218ba133\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:20.765136 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.765047 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bxpq\" (UniqueName: \"kubernetes.io/projected/96c645a0-f9e7-4bc9-893f-e3d2218ba133-kube-api-access-5bxpq\") pod \"kuadrant-operator-controller-manager-86878c769f-58lkk\" (UID: \"96c645a0-f9e7-4bc9-893f-e3d2218ba133\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:20.765318 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.765300 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96c645a0-f9e7-4bc9-893f-e3d2218ba133-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-58lkk\" (UID: \"96c645a0-f9e7-4bc9-893f-e3d2218ba133\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:20.774958 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.774934 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bxpq\" (UniqueName: \"kubernetes.io/projected/96c645a0-f9e7-4bc9-893f-e3d2218ba133-kube-api-access-5bxpq\") pod \"kuadrant-operator-controller-manager-86878c769f-58lkk\" (UID: \"96c645a0-f9e7-4bc9-893f-e3d2218ba133\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:20.943900 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:20.943819 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:21.088650 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:21.088615 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk"] Jul 07 16:01:21.092870 ip-10-0-128-121 kubenswrapper[2583]: W0707 16:01:21.092842 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c645a0_f9e7_4bc9_893f_e3d2218ba133.slice/crio-18182c797a62672fd2fc60a849f64eadee1ec757e886b856c24a666bdcc8359a WatchSource:0}: Error finding container 18182c797a62672fd2fc60a849f64eadee1ec757e886b856c24a666bdcc8359a: Status 404 returned error can't find the container with id 18182c797a62672fd2fc60a849f64eadee1ec757e886b856c24a666bdcc8359a Jul 07 16:01:21.407179 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:21.407149 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" event={"ID":"96c645a0-f9e7-4bc9-893f-e3d2218ba133","Type":"ContainerStarted","Data":"21509a0b79336b48325520d8dd117cc842ae957c7d64cac029ed6374176c76be"} Jul 07 16:01:21.407346 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:21.407205 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" event={"ID":"96c645a0-f9e7-4bc9-893f-e3d2218ba133","Type":"ContainerStarted","Data":"18182c797a62672fd2fc60a849f64eadee1ec757e886b856c24a666bdcc8359a"} Jul 07 16:01:21.407346 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:21.407305 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:21.433401 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:21.433350 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" podStartSLOduration=1.4333345 podStartE2EDuration="1.4333345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:01:21.430017555 +0000 UTC m=+1187.560430704" watchObservedRunningTime="2026-07-07 16:01:21.4333345 +0000 UTC m=+1187.563747646" Jul 07 16:01:32.414148 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:32.414112 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-58lkk" Jul 07 16:01:35.119397 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:35.119279 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:01:35.139755 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:01:35.121389 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:06:35.150609 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:06:35.150488 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:06:35.154160 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:06:35.152622 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:11:35.189939 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:11:35.189827 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:11:35.203339 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:11:35.192013 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:14:42.144925 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.144894 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp"] Jul 07 16:14:42.147664 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.147638 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.151344 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.151322 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-10313d\"/\"kube-root-ca.crt\"" Jul 07 16:14:42.152677 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.152573 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-10313d\"/\"test-model-10313d-kserve-self-signed-certs\"" Jul 07 16:14:42.152785 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.152752 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-10313d\"/\"openshift-service-ca.crt\"" Jul 07 16:14:42.152785 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.152774 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-10313d\"/\"default-dockercfg-7g22t\"" Jul 07 16:14:42.167186 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.167161 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp"] Jul 07 16:14:42.287292 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.287258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tls-certs\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.287469 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.287318 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-model-cache\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.287469 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.287389 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zpf4\" (UniqueName: \"kubernetes.io/projected/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-kube-api-access-6zpf4\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.287602 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.287463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-dshm\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.287602 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.287497 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-home\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.287602 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.287530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tmp-dir\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.389097 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.389060 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tls-certs\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.389282 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.389121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-model-cache\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.389282 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.389168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zpf4\" (UniqueName: \"kubernetes.io/projected/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-kube-api-access-6zpf4\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.389282 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.389195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-dshm\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.389282 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.389221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-home\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.389282 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.389251 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tmp-dir\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.389659 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.389632 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-model-cache\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.389785 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.389683 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-home\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.389785 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.389709 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tmp-dir\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.391411 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.391385 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-dshm\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.391742 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.391723 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tls-certs\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.401034 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.400985 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zpf4\" (UniqueName: \"kubernetes.io/projected/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-kube-api-access-6zpf4\") pod \"test-model-10313d-kserve-54789bc4c7-7p7sp\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.457314 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.457273 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:42.599986 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:42.599947 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp"] Jul 07 16:14:43.316853 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:43.316819 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" event={"ID":"cd1c6086-76e4-44a2-bb9e-f1345bc62f02","Type":"ContainerStarted","Data":"ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e"} Jul 07 16:14:43.317256 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:43.316859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" event={"ID":"cd1c6086-76e4-44a2-bb9e-f1345bc62f02","Type":"ContainerStarted","Data":"8a2c22beb82b4effe6267686515a0690b205ab55147dd8ecaa6c0bb402eae5e4"} Jul 07 16:14:43.338879 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:43.338832 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" podStartSLOduration=1.338817678 podStartE2EDuration="1.338817678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:14:43.337181389 +0000 UTC m=+1989.467594535" watchObservedRunningTime="2026-07-07 16:14:43.338817678 +0000 UTC m=+1989.469230824" Jul 07 16:14:52.458233 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:52.458193 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:52.458233 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:52.458236 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:52.470560 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:52.470523 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:14:53.375036 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:14:53.375001 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:15:37.954599 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:37.954567 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp"] Jul 07 16:15:52.974613 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:52.974536 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" podUID="cd1c6086-76e4-44a2-bb9e-f1345bc62f02" containerName="main" containerID="cri-o://ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e" gracePeriod=45 Jul 07 16:15:53.230038 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.229973 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:15:53.366277 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366234 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-model-cache\") pod \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " Jul 07 16:15:53.366277 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366281 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tmp-dir\") pod \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " Jul 07 16:15:53.366537 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366319 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-home\") pod \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " Jul 07 16:15:53.366537 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366405 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zpf4\" (UniqueName: \"kubernetes.io/projected/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-kube-api-access-6zpf4\") pod \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " Jul 07 16:15:53.366537 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366461 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tls-certs\") pod \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " Jul 07 16:15:53.366537 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366500 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-dshm\") pod \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\" (UID: \"cd1c6086-76e4-44a2-bb9e-f1345bc62f02\") " Jul 07 16:15:53.366777 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366570 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd1c6086-76e4-44a2-bb9e-f1345bc62f02" (UID: "cd1c6086-76e4-44a2-bb9e-f1345bc62f02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:15:53.366777 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366729 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-home" (OuterVolumeSpecName: "home") pod "cd1c6086-76e4-44a2-bb9e-f1345bc62f02" (UID: "cd1c6086-76e4-44a2-bb9e-f1345bc62f02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:15:53.366936 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366911 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:15:53.367069 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.366940 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:15:53.367069 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.367019 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd1c6086-76e4-44a2-bb9e-f1345bc62f02" (UID: "cd1c6086-76e4-44a2-bb9e-f1345bc62f02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:15:53.368896 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.368871 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-dshm" (OuterVolumeSpecName: "dshm") pod "cd1c6086-76e4-44a2-bb9e-f1345bc62f02" (UID: "cd1c6086-76e4-44a2-bb9e-f1345bc62f02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:15:53.369000 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.368888 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-kube-api-access-6zpf4" (OuterVolumeSpecName: "kube-api-access-6zpf4") pod "cd1c6086-76e4-44a2-bb9e-f1345bc62f02" (UID: "cd1c6086-76e4-44a2-bb9e-f1345bc62f02"). InnerVolumeSpecName "kube-api-access-6zpf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:15:53.369000 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.368914 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd1c6086-76e4-44a2-bb9e-f1345bc62f02" (UID: "cd1c6086-76e4-44a2-bb9e-f1345bc62f02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:15:53.468477 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.468440 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:15:53.468477 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.468469 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zpf4\" (UniqueName: \"kubernetes.io/projected/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-kube-api-access-6zpf4\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:15:53.468477 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.468480 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:15:53.468477 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.468489 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6086-76e4-44a2-bb9e-f1345bc62f02-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:15:53.659107 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.659071 2583 generic.go:356] "Generic (PLEG): container finished" podID="cd1c6086-76e4-44a2-bb9e-f1345bc62f02" containerID="ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e" exitCode=0 Jul 07 16:15:53.659286 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.659119 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" event={"ID":"cd1c6086-76e4-44a2-bb9e-f1345bc62f02","Type":"ContainerDied","Data":"ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e"} Jul 07 16:15:53.659286 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.659147 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" event={"ID":"cd1c6086-76e4-44a2-bb9e-f1345bc62f02","Type":"ContainerDied","Data":"8a2c22beb82b4effe6267686515a0690b205ab55147dd8ecaa6c0bb402eae5e4"} Jul 07 16:15:53.659286 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.659163 2583 scope.go:117] "RemoveContainer" containerID="ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e" Jul 07 16:15:53.659286 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.659119 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp" Jul 07 16:15:53.668177 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.668161 2583 scope.go:117] "RemoveContainer" containerID="ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e" Jul 07 16:15:53.668435 ip-10-0-128-121 kubenswrapper[2583]: E0707 16:15:53.668417 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e\": container with ID starting with ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e not found: ID does not exist" containerID="ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e" Jul 07 16:15:53.668483 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.668445 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e"} err="failed to get container status \"ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e\": rpc error: code = NotFound desc = could not find container \"ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e\": container with ID starting with ac50ca8e0bd13dbeb7192c718d697dd297576dac00a1ebf0c98c318855bf935e not found: ID does not exist" Jul 07 16:15:53.688008 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.687977 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp"] Jul 07 16:15:53.690168 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:53.690140 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-10313d/test-model-10313d-kserve-54789bc4c7-7p7sp"] Jul 07 16:15:54.798927 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:15:54.798889 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd1c6086-76e4-44a2-bb9e-f1345bc62f02" path="/var/lib/kubelet/pods/cd1c6086-76e4-44a2-bb9e-f1345bc62f02/volumes" Jul 07 16:16:35.220533 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:16:35.220421 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:16:35.227189 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:16:35.222679 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:17:46.623513 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.623478 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l"] Jul 07 16:17:46.623986 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.623973 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1c6086-76e4-44a2-bb9e-f1345bc62f02" containerName="main" Jul 07 16:17:46.624031 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.623989 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1c6086-76e4-44a2-bb9e-f1345bc62f02" containerName="main" Jul 07 16:17:46.624067 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.624055 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1c6086-76e4-44a2-bb9e-f1345bc62f02" containerName="main" Jul 07 16:17:46.626872 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.626855 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.630245 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.630225 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-382396\"/\"default-dockercfg-5rwxp\"" Jul 07 16:17:46.630441 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.630421 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-382396\"/\"kube-root-ca.crt\"" Jul 07 16:17:46.631366 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.631352 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-382396\"/\"auth-test-model-382396-kserve-self-signed-certs\"" Jul 07 16:17:46.631417 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.631403 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-382396\"/\"openshift-service-ca.crt\"" Jul 07 16:17:46.637705 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.637676 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l"] Jul 07 16:17:46.747536 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.747506 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-home\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.747720 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.747562 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdpzr\" (UniqueName: \"kubernetes.io/projected/de1098a1-d619-4f9b-adf7-9834c1d93cda-kube-api-access-mdpzr\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.747720 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.747650 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-model-cache\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.747720 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.747686 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-dshm\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.747720 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.747712 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1098a1-d619-4f9b-adf7-9834c1d93cda-tls-certs\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.747883 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.747781 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-tmp-dir\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.848557 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.848516 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-home\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.848734 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.848571 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdpzr\" (UniqueName: \"kubernetes.io/projected/de1098a1-d619-4f9b-adf7-9834c1d93cda-kube-api-access-mdpzr\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.848734 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.848605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-model-cache\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.848734 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.848623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-dshm\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.848734 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.848641 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1098a1-d619-4f9b-adf7-9834c1d93cda-tls-certs\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.848734 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.848679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-tmp-dir\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.849008 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.848885 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-home\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.849008 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.848952 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-model-cache\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.849113 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.849070 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-tmp-dir\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.850851 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.850826 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-dshm\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.851038 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.851022 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1098a1-d619-4f9b-adf7-9834c1d93cda-tls-certs\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.860380 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.860359 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdpzr\" (UniqueName: \"kubernetes.io/projected/de1098a1-d619-4f9b-adf7-9834c1d93cda-kube-api-access-mdpzr\") pod \"auth-test-model-382396-kserve-6cccf97df9-h9q7l\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:46.937276 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:46.937218 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:47.078156 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:47.078124 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l"] Jul 07 16:17:47.081995 ip-10-0-128-121 kubenswrapper[2583]: W0707 16:17:47.081974 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde1098a1_d619_4f9b_adf7_9834c1d93cda.slice/crio-52e59b50401e02d63226d714835dfece40093bfb93bfe8f99f217ef78f1b08e1 WatchSource:0}: Error finding container 52e59b50401e02d63226d714835dfece40093bfb93bfe8f99f217ef78f1b08e1: Status 404 returned error can't find the container with id 52e59b50401e02d63226d714835dfece40093bfb93bfe8f99f217ef78f1b08e1 Jul 07 16:17:47.215771 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:47.215658 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" event={"ID":"de1098a1-d619-4f9b-adf7-9834c1d93cda","Type":"ContainerStarted","Data":"59145eac8c9ed336110b793bb82f4aae96e20e895e1ac38969d11f9027290b4c"} Jul 07 16:17:47.215771 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:47.215699 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" event={"ID":"de1098a1-d619-4f9b-adf7-9834c1d93cda","Type":"ContainerStarted","Data":"52e59b50401e02d63226d714835dfece40093bfb93bfe8f99f217ef78f1b08e1"} Jul 07 16:17:47.239603 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:47.239537 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" podStartSLOduration=1.239523328 podStartE2EDuration="1.239523328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:17:47.239189131 +0000 UTC m=+2173.369602278" watchObservedRunningTime="2026-07-07 16:17:47.239523328 +0000 UTC m=+2173.369936474" Jul 07 16:17:56.937437 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:56.937398 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:56.937437 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:56.937441 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:56.950229 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:56.950204 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:17:57.276181 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:17:57.276091 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:20:28.854829 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:28.854742 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l"] Jul 07 16:20:37.923935 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:37.923901 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm"] Jul 07 16:20:37.928619 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:37.928597 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:37.931462 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:37.931437 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-382396\"/\"sub-test-model-382396-kserve-self-signed-certs\"" Jul 07 16:20:37.938569 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:37.938532 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm"] Jul 07 16:20:38.083100 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.083055 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-model-cache\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.083303 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.083125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72m4l\" (UniqueName: \"kubernetes.io/projected/582ebac6-568d-4a8f-ab73-9f328acba050-kube-api-access-72m4l\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.083303 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.083179 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582ebac6-568d-4a8f-ab73-9f328acba050-tls-certs\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.083303 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.083205 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-tmp-dir\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.083303 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.083241 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-home\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.083303 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.083255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-dshm\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.184108 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.184018 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-home\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.184108 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.184058 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-dshm\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.184311 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.184123 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-model-cache\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.184311 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.184183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72m4l\" (UniqueName: \"kubernetes.io/projected/582ebac6-568d-4a8f-ab73-9f328acba050-kube-api-access-72m4l\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.184311 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.184235 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582ebac6-568d-4a8f-ab73-9f328acba050-tls-certs\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.184311 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.184271 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-tmp-dir\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.184513 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.184458 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-home\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.184593 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.184536 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-model-cache\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.184726 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.184701 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-tmp-dir\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.186314 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.186288 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-dshm\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.186712 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.186692 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582ebac6-568d-4a8f-ab73-9f328acba050-tls-certs\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.193796 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.193776 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72m4l\" (UniqueName: \"kubernetes.io/projected/582ebac6-568d-4a8f-ab73-9f328acba050-kube-api-access-72m4l\") pod \"sub-test-model-382396-kserve-79c9d5ff97-br9tm\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.238814 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.238786 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:38.372613 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:38.372588 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm"] Jul 07 16:20:38.374886 ip-10-0-128-121 kubenswrapper[2583]: W0707 16:20:38.374863 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod582ebac6_568d_4a8f_ab73_9f328acba050.slice/crio-df0d85cdc88d0ff2b406bd96d33098163c6f6da64489d70218f8eddc415c5de4 WatchSource:0}: Error finding container df0d85cdc88d0ff2b406bd96d33098163c6f6da64489d70218f8eddc415c5de4: Status 404 returned error can't find the container with id df0d85cdc88d0ff2b406bd96d33098163c6f6da64489d70218f8eddc415c5de4 Jul 07 16:20:39.051178 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:39.051137 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" event={"ID":"582ebac6-568d-4a8f-ab73-9f328acba050","Type":"ContainerStarted","Data":"bb7bf6a853a972839d20fd8977398c19ccf223eebd74a49a32d05ae8ff5dd419"} Jul 07 16:20:39.051648 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:39.051190 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" event={"ID":"582ebac6-568d-4a8f-ab73-9f328acba050","Type":"ContainerStarted","Data":"df0d85cdc88d0ff2b406bd96d33098163c6f6da64489d70218f8eddc415c5de4"} Jul 07 16:20:39.071972 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:39.071916 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" podStartSLOduration=2.071897319 podStartE2EDuration="2.071897319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:20:39.070467288 +0000 UTC m=+2345.200880434" watchObservedRunningTime="2026-07-07 16:20:39.071897319 +0000 UTC m=+2345.202310470" Jul 07 16:20:43.875427 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:43.875385 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" podUID="de1098a1-d619-4f9b-adf7-9834c1d93cda" containerName="main" containerID="cri-o://59145eac8c9ed336110b793bb82f4aae96e20e895e1ac38969d11f9027290b4c" gracePeriod=45 Jul 07 16:20:44.078639 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.078610 2583 generic.go:356] "Generic (PLEG): container finished" podID="de1098a1-d619-4f9b-adf7-9834c1d93cda" containerID="59145eac8c9ed336110b793bb82f4aae96e20e895e1ac38969d11f9027290b4c" exitCode=0 Jul 07 16:20:44.078786 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.078689 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" event={"ID":"de1098a1-d619-4f9b-adf7-9834c1d93cda","Type":"ContainerDied","Data":"59145eac8c9ed336110b793bb82f4aae96e20e895e1ac38969d11f9027290b4c"} Jul 07 16:20:44.132728 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.132675 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:20:44.238646 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.238598 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-tmp-dir\") pod \"de1098a1-d619-4f9b-adf7-9834c1d93cda\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " Jul 07 16:20:44.238646 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.238653 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-home\") pod \"de1098a1-d619-4f9b-adf7-9834c1d93cda\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " Jul 07 16:20:44.238887 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.238687 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-model-cache\") pod \"de1098a1-d619-4f9b-adf7-9834c1d93cda\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " Jul 07 16:20:44.238887 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.238732 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-dshm\") pod \"de1098a1-d619-4f9b-adf7-9834c1d93cda\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " Jul 07 16:20:44.238887 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.238756 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1098a1-d619-4f9b-adf7-9834c1d93cda-tls-certs\") pod \"de1098a1-d619-4f9b-adf7-9834c1d93cda\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " Jul 07 16:20:44.238887 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.238797 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdpzr\" (UniqueName: \"kubernetes.io/projected/de1098a1-d619-4f9b-adf7-9834c1d93cda-kube-api-access-mdpzr\") pod \"de1098a1-d619-4f9b-adf7-9834c1d93cda\" (UID: \"de1098a1-d619-4f9b-adf7-9834c1d93cda\") " Jul 07 16:20:44.239106 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.238921 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de1098a1-d619-4f9b-adf7-9834c1d93cda" (UID: "de1098a1-d619-4f9b-adf7-9834c1d93cda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:20:44.239106 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.238977 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-home" (OuterVolumeSpecName: "home") pod "de1098a1-d619-4f9b-adf7-9834c1d93cda" (UID: "de1098a1-d619-4f9b-adf7-9834c1d93cda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:20:44.239106 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.238986 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-model-cache" (OuterVolumeSpecName: "model-cache") pod "de1098a1-d619-4f9b-adf7-9834c1d93cda" (UID: "de1098a1-d619-4f9b-adf7-9834c1d93cda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:20:44.239231 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.239157 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:20:44.239231 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.239171 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:20:44.239231 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.239180 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:20:44.240924 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.240894 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-dshm" (OuterVolumeSpecName: "dshm") pod "de1098a1-d619-4f9b-adf7-9834c1d93cda" (UID: "de1098a1-d619-4f9b-adf7-9834c1d93cda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:20:44.241022 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.240966 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1098a1-d619-4f9b-adf7-9834c1d93cda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de1098a1-d619-4f9b-adf7-9834c1d93cda" (UID: "de1098a1-d619-4f9b-adf7-9834c1d93cda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:20:44.241070 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.241049 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1098a1-d619-4f9b-adf7-9834c1d93cda-kube-api-access-mdpzr" (OuterVolumeSpecName: "kube-api-access-mdpzr") pod "de1098a1-d619-4f9b-adf7-9834c1d93cda" (UID: "de1098a1-d619-4f9b-adf7-9834c1d93cda"). InnerVolumeSpecName "kube-api-access-mdpzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:20:44.340261 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.340225 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1098a1-d619-4f9b-adf7-9834c1d93cda-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:20:44.340261 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.340253 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1098a1-d619-4f9b-adf7-9834c1d93cda-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:20:44.340261 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:44.340264 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdpzr\" (UniqueName: \"kubernetes.io/projected/de1098a1-d619-4f9b-adf7-9834c1d93cda-kube-api-access-mdpzr\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:20:45.084980 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:45.084891 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" event={"ID":"de1098a1-d619-4f9b-adf7-9834c1d93cda","Type":"ContainerDied","Data":"52e59b50401e02d63226d714835dfece40093bfb93bfe8f99f217ef78f1b08e1"} Jul 07 16:20:45.084980 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:45.084921 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l" Jul 07 16:20:45.084980 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:45.084938 2583 scope.go:117] "RemoveContainer" containerID="59145eac8c9ed336110b793bb82f4aae96e20e895e1ac38969d11f9027290b4c" Jul 07 16:20:45.104962 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:45.104936 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l"] Jul 07 16:20:45.109210 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:45.109189 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-382396/auth-test-model-382396-kserve-6cccf97df9-h9q7l"] Jul 07 16:20:46.799004 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:46.798974 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de1098a1-d619-4f9b-adf7-9834c1d93cda" path="/var/lib/kubelet/pods/de1098a1-d619-4f9b-adf7-9834c1d93cda/volumes" Jul 07 16:20:48.239646 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:48.239600 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:48.240061 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:48.239664 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:48.252421 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:48.252391 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:20:49.117145 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:20:49.117114 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:21:35.251337 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:21:35.251226 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:21:35.259841 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:21:35.254204 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzqpt_3b0053d1-a54c-4d0a-8b90-e37baf41c62f/ovn-acl-logging/0.log" Jul 07 16:21:47.240633 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:21:47.240586 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm"] Jul 07 16:22:02.260456 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.260335 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" podUID="582ebac6-568d-4a8f-ab73-9f328acba050" containerName="main" containerID="cri-o://bb7bf6a853a972839d20fd8977398c19ccf223eebd74a49a32d05ae8ff5dd419" gracePeriod=45 Jul 07 16:22:02.474002 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.473972 2583 generic.go:356] "Generic (PLEG): container finished" podID="582ebac6-568d-4a8f-ab73-9f328acba050" containerID="bb7bf6a853a972839d20fd8977398c19ccf223eebd74a49a32d05ae8ff5dd419" exitCode=0 Jul 07 16:22:02.474167 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.474047 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" event={"ID":"582ebac6-568d-4a8f-ab73-9f328acba050","Type":"ContainerDied","Data":"bb7bf6a853a972839d20fd8977398c19ccf223eebd74a49a32d05ae8ff5dd419"} Jul 07 16:22:02.523655 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.523597 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:22:02.704977 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.704941 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-dshm\") pod \"582ebac6-568d-4a8f-ab73-9f328acba050\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " Jul 07 16:22:02.704977 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.704981 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-home\") pod \"582ebac6-568d-4a8f-ab73-9f328acba050\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " Jul 07 16:22:02.705176 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.705022 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72m4l\" (UniqueName: \"kubernetes.io/projected/582ebac6-568d-4a8f-ab73-9f328acba050-kube-api-access-72m4l\") pod \"582ebac6-568d-4a8f-ab73-9f328acba050\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " Jul 07 16:22:02.705176 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.705057 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582ebac6-568d-4a8f-ab73-9f328acba050-tls-certs\") pod \"582ebac6-568d-4a8f-ab73-9f328acba050\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " Jul 07 16:22:02.705176 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.705089 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-model-cache\") pod \"582ebac6-568d-4a8f-ab73-9f328acba050\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " Jul 07 16:22:02.705293 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.705231 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-tmp-dir\") pod \"582ebac6-568d-4a8f-ab73-9f328acba050\" (UID: \"582ebac6-568d-4a8f-ab73-9f328acba050\") " Jul 07 16:22:02.705354 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.705299 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-home" (OuterVolumeSpecName: "home") pod "582ebac6-568d-4a8f-ab73-9f328acba050" (UID: "582ebac6-568d-4a8f-ab73-9f328acba050"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:22:02.705446 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.705417 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-model-cache" (OuterVolumeSpecName: "model-cache") pod "582ebac6-568d-4a8f-ab73-9f328acba050" (UID: "582ebac6-568d-4a8f-ab73-9f328acba050"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:22:02.705605 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.705498 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "582ebac6-568d-4a8f-ab73-9f328acba050" (UID: "582ebac6-568d-4a8f-ab73-9f328acba050"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:22:02.705680 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.705606 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:22:02.705680 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.705619 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:22:02.707295 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.707268 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582ebac6-568d-4a8f-ab73-9f328acba050-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "582ebac6-568d-4a8f-ab73-9f328acba050" (UID: "582ebac6-568d-4a8f-ab73-9f328acba050"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:22:02.707295 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.707283 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582ebac6-568d-4a8f-ab73-9f328acba050-kube-api-access-72m4l" (OuterVolumeSpecName: "kube-api-access-72m4l") pod "582ebac6-568d-4a8f-ab73-9f328acba050" (UID: "582ebac6-568d-4a8f-ab73-9f328acba050"). InnerVolumeSpecName "kube-api-access-72m4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:22:02.707414 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.707329 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-dshm" (OuterVolumeSpecName: "dshm") pod "582ebac6-568d-4a8f-ab73-9f328acba050" (UID: "582ebac6-568d-4a8f-ab73-9f328acba050"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:22:02.806338 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.806303 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582ebac6-568d-4a8f-ab73-9f328acba050-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:22:02.806338 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.806337 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:22:02.806522 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.806351 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582ebac6-568d-4a8f-ab73-9f328acba050-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:22:02.806522 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:02.806366 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72m4l\" (UniqueName: \"kubernetes.io/projected/582ebac6-568d-4a8f-ab73-9f328acba050-kube-api-access-72m4l\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:22:03.487451 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:03.487415 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" event={"ID":"582ebac6-568d-4a8f-ab73-9f328acba050","Type":"ContainerDied","Data":"df0d85cdc88d0ff2b406bd96d33098163c6f6da64489d70218f8eddc415c5de4"} Jul 07 16:22:03.487451 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:03.487435 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm" Jul 07 16:22:03.487976 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:03.487466 2583 scope.go:117] "RemoveContainer" containerID="bb7bf6a853a972839d20fd8977398c19ccf223eebd74a49a32d05ae8ff5dd419" Jul 07 16:22:03.508634 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:03.508606 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm"] Jul 07 16:22:03.513520 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:03.513495 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-382396/sub-test-model-382396-kserve-79c9d5ff97-br9tm"] Jul 07 16:22:04.798237 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:04.798205 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="582ebac6-568d-4a8f-ab73-9f328acba050" path="/var/lib/kubelet/pods/582ebac6-568d-4a8f-ab73-9f328acba050/volumes" Jul 07 16:22:15.921884 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.921849 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28"] Jul 07 16:22:15.922431 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.922257 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1098a1-d619-4f9b-adf7-9834c1d93cda" containerName="main" Jul 07 16:22:15.922431 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.922268 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1098a1-d619-4f9b-adf7-9834c1d93cda" containerName="main" Jul 07 16:22:15.922431 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.922280 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582ebac6-568d-4a8f-ab73-9f328acba050" containerName="main" Jul 07 16:22:15.922431 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.922285 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="582ebac6-568d-4a8f-ab73-9f328acba050" containerName="main" Jul 07 16:22:15.922431 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.922336 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="582ebac6-568d-4a8f-ab73-9f328acba050" containerName="main" Jul 07 16:22:15.922431 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.922348 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1098a1-d619-4f9b-adf7-9834c1d93cda" containerName="main" Jul 07 16:22:15.924397 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.924377 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:15.928450 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.928424 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"default-dockercfg-zvrcp\"" Jul 07 16:22:15.928598 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.928449 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"openshift-service-ca.crt\"" Jul 07 16:22:15.928598 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.928458 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"kube-root-ca.crt\"" Jul 07 16:22:15.928598 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.928484 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"rate-test-model-17e556-kserve-self-signed-certs\"" Jul 07 16:22:15.935692 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:15.935669 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28"] Jul 07 16:22:16.035427 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.035399 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-home\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.035644 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.035445 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-dshm\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.035644 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.035470 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ls6j\" (UniqueName: \"kubernetes.io/projected/cad0c92b-4b6a-405c-a075-93890a77ac16-kube-api-access-7ls6j\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.035644 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.035527 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-model-cache\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.035644 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.035575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad0c92b-4b6a-405c-a075-93890a77ac16-tls-certs\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.035644 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.035593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-tmp-dir\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.136947 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.136915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-home\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.137157 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.136976 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-dshm\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.137157 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.137012 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ls6j\" (UniqueName: \"kubernetes.io/projected/cad0c92b-4b6a-405c-a075-93890a77ac16-kube-api-access-7ls6j\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.137157 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.137063 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-model-cache\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.137157 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.137108 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad0c92b-4b6a-405c-a075-93890a77ac16-tls-certs\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.137157 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.137136 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-tmp-dir\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.137438 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.137382 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-home\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.137438 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.137415 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-model-cache\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.137539 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.137477 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-tmp-dir\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.139282 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.139256 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-dshm\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.139521 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.139504 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad0c92b-4b6a-405c-a075-93890a77ac16-tls-certs\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.147248 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.147225 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ls6j\" (UniqueName: \"kubernetes.io/projected/cad0c92b-4b6a-405c-a075-93890a77ac16-kube-api-access-7ls6j\") pod \"rate-test-model-17e556-kserve-dbdb76479-4nm28\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.234726 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.234656 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:16.368876 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.368850 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28"] Jul 07 16:22:16.370948 ip-10-0-128-121 kubenswrapper[2583]: W0707 16:22:16.370914 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcad0c92b_4b6a_405c_a075_93890a77ac16.slice/crio-aadd713931cc2b0a44ef7ac87ceb9de8d44ace4a8f81c824c3b984a2c0164a00 WatchSource:0}: Error finding container aadd713931cc2b0a44ef7ac87ceb9de8d44ace4a8f81c824c3b984a2c0164a00: Status 404 returned error can't find the container with id aadd713931cc2b0a44ef7ac87ceb9de8d44ace4a8f81c824c3b984a2c0164a00 Jul 07 16:22:16.555981 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.555892 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" event={"ID":"cad0c92b-4b6a-405c-a075-93890a77ac16","Type":"ContainerStarted","Data":"f7e759f7012fdceacd2df23274d6d6c45b6170c848a032c744a26f5d47218945"} Jul 07 16:22:16.555981 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.555934 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" event={"ID":"cad0c92b-4b6a-405c-a075-93890a77ac16","Type":"ContainerStarted","Data":"aadd713931cc2b0a44ef7ac87ceb9de8d44ace4a8f81c824c3b984a2c0164a00"} Jul 07 16:22:16.576977 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:16.576923 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" podStartSLOduration=1.576904929 podStartE2EDuration="1.576904929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:22:16.576498068 +0000 UTC m=+2442.706911214" watchObservedRunningTime="2026-07-07 16:22:16.576904929 +0000 UTC m=+2442.707318076" Jul 07 16:22:26.234937 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:26.234901 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:26.235361 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:26.235073 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:26.247125 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:26.247104 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:22:26.616160 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:22:26.616130 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:23:23.741269 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:23.741237 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28"] Jul 07 16:23:38.761101 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:38.761009 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" podUID="cad0c92b-4b6a-405c-a075-93890a77ac16" containerName="main" containerID="cri-o://f7e759f7012fdceacd2df23274d6d6c45b6170c848a032c744a26f5d47218945" gracePeriod=45 Jul 07 16:23:38.958517 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:38.958476 2583 generic.go:356] "Generic (PLEG): container finished" podID="cad0c92b-4b6a-405c-a075-93890a77ac16" containerID="f7e759f7012fdceacd2df23274d6d6c45b6170c848a032c744a26f5d47218945" exitCode=0 Jul 07 16:23:38.958676 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:38.958568 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" event={"ID":"cad0c92b-4b6a-405c-a075-93890a77ac16","Type":"ContainerDied","Data":"f7e759f7012fdceacd2df23274d6d6c45b6170c848a032c744a26f5d47218945"} Jul 07 16:23:39.007628 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.007602 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:23:39.125485 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.125449 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-model-cache\") pod \"cad0c92b-4b6a-405c-a075-93890a77ac16\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " Jul 07 16:23:39.125485 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.125483 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad0c92b-4b6a-405c-a075-93890a77ac16-tls-certs\") pod \"cad0c92b-4b6a-405c-a075-93890a77ac16\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " Jul 07 16:23:39.125761 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.125519 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-dshm\") pod \"cad0c92b-4b6a-405c-a075-93890a77ac16\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " Jul 07 16:23:39.125761 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.125567 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-home\") pod \"cad0c92b-4b6a-405c-a075-93890a77ac16\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " Jul 07 16:23:39.125761 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.125596 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ls6j\" (UniqueName: \"kubernetes.io/projected/cad0c92b-4b6a-405c-a075-93890a77ac16-kube-api-access-7ls6j\") pod \"cad0c92b-4b6a-405c-a075-93890a77ac16\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " Jul 07 16:23:39.125761 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.125715 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-tmp-dir\") pod \"cad0c92b-4b6a-405c-a075-93890a77ac16\" (UID: \"cad0c92b-4b6a-405c-a075-93890a77ac16\") " Jul 07 16:23:39.125989 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.125819 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-model-cache" (OuterVolumeSpecName: "model-cache") pod "cad0c92b-4b6a-405c-a075-93890a77ac16" (UID: "cad0c92b-4b6a-405c-a075-93890a77ac16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:23:39.125989 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.125835 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-home" (OuterVolumeSpecName: "home") pod "cad0c92b-4b6a-405c-a075-93890a77ac16" (UID: "cad0c92b-4b6a-405c-a075-93890a77ac16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:23:39.125989 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.125966 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cad0c92b-4b6a-405c-a075-93890a77ac16" (UID: "cad0c92b-4b6a-405c-a075-93890a77ac16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:23:39.126120 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.126106 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:23:39.126173 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.126123 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:23:39.126173 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.126150 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:23:39.127908 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.127881 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad0c92b-4b6a-405c-a075-93890a77ac16-kube-api-access-7ls6j" (OuterVolumeSpecName: "kube-api-access-7ls6j") pod "cad0c92b-4b6a-405c-a075-93890a77ac16" (UID: "cad0c92b-4b6a-405c-a075-93890a77ac16"). InnerVolumeSpecName "kube-api-access-7ls6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:23:39.128125 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.128109 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-dshm" (OuterVolumeSpecName: "dshm") pod "cad0c92b-4b6a-405c-a075-93890a77ac16" (UID: "cad0c92b-4b6a-405c-a075-93890a77ac16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:23:39.128179 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.128122 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad0c92b-4b6a-405c-a075-93890a77ac16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cad0c92b-4b6a-405c-a075-93890a77ac16" (UID: "cad0c92b-4b6a-405c-a075-93890a77ac16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:23:39.227680 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.227647 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad0c92b-4b6a-405c-a075-93890a77ac16-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:23:39.227680 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.227673 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad0c92b-4b6a-405c-a075-93890a77ac16-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:23:39.227680 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.227682 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ls6j\" (UniqueName: \"kubernetes.io/projected/cad0c92b-4b6a-405c-a075-93890a77ac16-kube-api-access-7ls6j\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\"" Jul 07 16:23:39.964643 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.964607 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" event={"ID":"cad0c92b-4b6a-405c-a075-93890a77ac16","Type":"ContainerDied","Data":"aadd713931cc2b0a44ef7ac87ceb9de8d44ace4a8f81c824c3b984a2c0164a00"} Jul 07 16:23:39.965074 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.964659 2583 scope.go:117] "RemoveContainer" containerID="f7e759f7012fdceacd2df23274d6d6c45b6170c848a032c744a26f5d47218945" Jul 07 16:23:39.965074 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.964663 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28" Jul 07 16:23:39.988318 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.988291 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28"] Jul 07 16:23:39.993082 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:39.993061 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/rate-test-model-17e556-kserve-dbdb76479-4nm28"] Jul 07 16:23:40.798062 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:23:40.798020 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad0c92b-4b6a-405c-a075-93890a77ac16" path="/var/lib/kubelet/pods/cad0c92b-4b6a-405c-a075-93890a77ac16/volumes" Jul 07 16:24:22.701935 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:22.701905 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685dff6fcf-ghv85_3c3503fe-bc75-4ace-a40c-af18bac782e9/copy-manifests/0.log" Jul 07 16:24:22.711252 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:22.711235 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685dff6fcf-ghv85_3c3503fe-bc75-4ace-a40c-af18bac782e9/manager/0.log" Jul 07 16:24:24.634678 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:24.634651 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rjdrw_031f4fa5-906a-40dc-baee-6d1b84859fb7/kuadrant-console-plugin/0.log" Jul 07 16:24:24.885256 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:24.885175 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-58lkk_96c645a0-f9e7-4bc9-893f-e3d2218ba133/manager/0.log" Jul 07 16:24:25.744467 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:25.744432 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ccc59c597-hdv58_1f958698-a875-466d-af7f-3a4e6c6528eb/kube-auth-proxy/0.log" Jul 07 16:24:26.107148 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:26.107119 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-r7ldp_b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a/payload-pre-processing/0.log" Jul 07 16:24:26.224985 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:26.224959 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-ctm2s_cbc19cca-d26f-4954-ba6b-c524423efd2f/payload-processing/0.log" Jul 07 16:24:26.942977 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:26.942948 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-bkl52_f555cd03-9179-4f54-af32-b4c00d95be92/main/0.log" Jul 07 16:24:27.297562 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:27.297465 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-xkt7x_486b7d60-f27f-48df-a78a-63afc74dca44/main/0.log" Jul 07 16:24:34.806818 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:34.806793 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mpk6m_35748986-3b58-45b7-a1e0-a77c803c1ad3/global-pull-secret-syncer/0.log" Jul 07 16:24:34.984909 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:34.984832 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s86zr_08452baf-e640-4628-8782-25b1cdb96215/konnectivity-agent/0.log" Jul 07 16:24:35.008387 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:35.008367 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-121.ec2.internal_dbf149aa0654ab8e90cfec7925c91095/haproxy/0.log" Jul 07 16:24:40.145291 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:40.145263 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rjdrw_031f4fa5-906a-40dc-baee-6d1b84859fb7/kuadrant-console-plugin/0.log" Jul 07 16:24:40.368532 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:40.368494 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-58lkk_96c645a0-f9e7-4bc9-893f-e3d2218ba133/manager/0.log" Jul 07 16:24:42.131103 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.131079 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70f7f655-a2c4-4409-90d8-146d74a5f46c/alertmanager/0.log" Jul 07 16:24:42.162167 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.162143 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70f7f655-a2c4-4409-90d8-146d74a5f46c/config-reloader/0.log" Jul 07 16:24:42.200160 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.200134 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70f7f655-a2c4-4409-90d8-146d74a5f46c/kube-rbac-proxy-web/0.log" Jul 07 16:24:42.235193 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.235146 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70f7f655-a2c4-4409-90d8-146d74a5f46c/kube-rbac-proxy/0.log" Jul 07 16:24:42.280501 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.280472 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70f7f655-a2c4-4409-90d8-146d74a5f46c/kube-rbac-proxy-metric/0.log" Jul 07 16:24:42.312718 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.312694 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70f7f655-a2c4-4409-90d8-146d74a5f46c/prom-label-proxy/0.log" Jul 07 16:24:42.347598 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.347569 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70f7f655-a2c4-4409-90d8-146d74a5f46c/init-config-reloader/0.log" Jul 07 16:24:42.435444 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.435418 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dnsvn_bf96a278-8b70-4c98-92f7-06739a72140a/kube-state-metrics/0.log" Jul 07 16:24:42.462272 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.462246 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dnsvn_bf96a278-8b70-4c98-92f7-06739a72140a/kube-rbac-proxy-main/0.log" Jul 07 16:24:42.497605 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.497511 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dnsvn_bf96a278-8b70-4c98-92f7-06739a72140a/kube-rbac-proxy-self/0.log" Jul 07 16:24:42.531158 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.531130 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5888f59b74-85v4h_2b1cbc78-5d52-4726-9b93-9bf60debadd9/metrics-server/0.log" Jul 07 16:24:42.563941 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.563917 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-f7d5t_940fa5fe-ff38-4d5f-ab3f-2b0cc0cc2ef9/monitoring-plugin/0.log" Jul 07 16:24:42.797428 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.797351 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mmfc2_763adb88-1e13-496b-919c-f7aa8c98746d/node-exporter/0.log" Jul 07 16:24:42.823073 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.823052 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mmfc2_763adb88-1e13-496b-919c-f7aa8c98746d/kube-rbac-proxy/0.log" Jul 07 16:24:42.853588 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.853570 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mmfc2_763adb88-1e13-496b-919c-f7aa8c98746d/init-textfile/0.log" Jul 07 16:24:42.891748 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.891725 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-pwh8d_caa73cf8-abd2-44ce-b682-e756427829de/kube-rbac-proxy-main/0.log" Jul 07 16:24:42.920444 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.920415 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-pwh8d_caa73cf8-abd2-44ce-b682-e756427829de/kube-rbac-proxy-self/0.log" Jul 07 16:24:42.948874 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.948849 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-pwh8d_caa73cf8-abd2-44ce-b682-e756427829de/openshift-state-metrics/0.log" Jul 07 16:24:42.992502 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:42.992482 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a68f7a5-a2df-41a0-8658-ab4cd6b91f40/prometheus/0.log" Jul 07 16:24:43.018333 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.018313 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a68f7a5-a2df-41a0-8658-ab4cd6b91f40/config-reloader/0.log" Jul 07 16:24:43.049241 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.049182 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a68f7a5-a2df-41a0-8658-ab4cd6b91f40/thanos-sidecar/0.log" Jul 07 16:24:43.077441 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.077417 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a68f7a5-a2df-41a0-8658-ab4cd6b91f40/kube-rbac-proxy-web/0.log" Jul 07 16:24:43.108380 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.108360 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a68f7a5-a2df-41a0-8658-ab4cd6b91f40/kube-rbac-proxy/0.log" Jul 07 16:24:43.135691 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.135666 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a68f7a5-a2df-41a0-8658-ab4cd6b91f40/kube-rbac-proxy-thanos/0.log" Jul 07 16:24:43.164650 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.164629 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a68f7a5-a2df-41a0-8658-ab4cd6b91f40/init-config-reloader/0.log" Jul 07 16:24:43.203494 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.203469 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-bk4br_bed3c459-1a4f-4517-9939-ac95d82c1dc1/prometheus-operator/0.log" Jul 07 16:24:43.231259 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.231236 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-bk4br_bed3c459-1a4f-4517-9939-ac95d82c1dc1/kube-rbac-proxy/0.log" Jul 07 16:24:43.293140 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.293113 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f49554c84-w4tmd_0721b254-5812-4a3f-878e-cf0b8ec01df5/telemeter-client/0.log" Jul 07 16:24:43.324369 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.324344 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f49554c84-w4tmd_0721b254-5812-4a3f-878e-cf0b8ec01df5/reload/0.log" Jul 07 16:24:43.351725 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.351704 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f49554c84-w4tmd_0721b254-5812-4a3f-878e-cf0b8ec01df5/kube-rbac-proxy/0.log" Jul 07 16:24:43.386938 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.386916 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f89874c98-95w2h_b2c34f0d-e299-4a0e-a693-08541fd93d20/thanos-query/0.log" Jul 07 16:24:43.421222 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.421201 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f89874c98-95w2h_b2c34f0d-e299-4a0e-a693-08541fd93d20/kube-rbac-proxy-web/0.log" Jul 07 16:24:43.452582 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.452540 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f89874c98-95w2h_b2c34f0d-e299-4a0e-a693-08541fd93d20/kube-rbac-proxy/0.log" Jul 07 16:24:43.479457 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.479437 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f89874c98-95w2h_b2c34f0d-e299-4a0e-a693-08541fd93d20/prom-label-proxy/0.log" Jul 07 16:24:43.506591 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.506575 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f89874c98-95w2h_b2c34f0d-e299-4a0e-a693-08541fd93d20/kube-rbac-proxy-rules/0.log" Jul 07 16:24:43.548877 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:43.548855 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f89874c98-95w2h_b2c34f0d-e299-4a0e-a693-08541fd93d20/kube-rbac-proxy-metrics/0.log" Jul 07 16:24:44.670772 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.670737 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw"] Jul 07 16:24:44.671223 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.671142 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad0c92b-4b6a-405c-a075-93890a77ac16" containerName="main" Jul 07 16:24:44.671223 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.671157 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad0c92b-4b6a-405c-a075-93890a77ac16" containerName="main" Jul 07 16:24:44.671427 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.671263 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad0c92b-4b6a-405c-a075-93890a77ac16" containerName="main" Jul 07 16:24:44.674171 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.674154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.676817 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.676799 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kfm9g\"/\"default-dockercfg-pc4lw\"" Jul 07 16:24:44.677835 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.677821 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfm9g\"/\"openshift-service-ca.crt\"" Jul 07 16:24:44.677902 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.677823 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfm9g\"/\"kube-root-ca.crt\"" Jul 07 16:24:44.684140 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.684120 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw"] Jul 07 16:24:44.724055 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.724027 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsgxx\" (UniqueName: \"kubernetes.io/projected/3cef10e7-3f96-47e5-a0b2-707ce99f2568-kube-api-access-xsgxx\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.724162 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.724074 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-proc\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.724162 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.724144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-sys\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.724239 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.724190 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-lib-modules\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.724239 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.724226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-podres\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.824889 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.824858 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-lib-modules\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.825041 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.824933 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-podres\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.825041 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.824965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsgxx\" (UniqueName: \"kubernetes.io/projected/3cef10e7-3f96-47e5-a0b2-707ce99f2568-kube-api-access-xsgxx\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.825041 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.825007 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-lib-modules\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.825041 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.825037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-proc\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.825181 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.825079 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-podres\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.825181 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.825100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-sys\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.825181 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.825123 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-proc\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.825285 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.825200 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cef10e7-3f96-47e5-a0b2-707ce99f2568-sys\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.836252 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.836219 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsgxx\" (UniqueName: \"kubernetes.io/projected/3cef10e7-3f96-47e5-a0b2-707ce99f2568-kube-api-access-xsgxx\") pod \"perf-node-gather-daemonset-qslmw\" (UID: \"3cef10e7-3f96-47e5-a0b2-707ce99f2568\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:44.985736 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:44.985662 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:45.120131 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:45.120107 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw"] Jul 07 16:24:45.122180 ip-10-0-128-121 kubenswrapper[2583]: W0707 16:24:45.122145 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3cef10e7_3f96_47e5_a0b2_707ce99f2568.slice/crio-d19bfc3b3c13bc3944c47af3dae3fa37d5a4a141be327aa34322f4977e839477 WatchSource:0}: Error finding container d19bfc3b3c13bc3944c47af3dae3fa37d5a4a141be327aa34322f4977e839477: Status 404 returned error can't find the container with id d19bfc3b3c13bc3944c47af3dae3fa37d5a4a141be327aa34322f4977e839477 Jul 07 16:24:45.280874 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:45.280800 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" event={"ID":"3cef10e7-3f96-47e5-a0b2-707ce99f2568","Type":"ContainerStarted","Data":"54eea55d253f276a69987c44a9ef1fd56dfeee276bb60c163506da297942420e"} Jul 07 16:24:45.280874 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:45.280852 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" event={"ID":"3cef10e7-3f96-47e5-a0b2-707ce99f2568","Type":"ContainerStarted","Data":"d19bfc3b3c13bc3944c47af3dae3fa37d5a4a141be327aa34322f4977e839477"} Jul 07 16:24:45.281041 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:45.280916 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:45.300396 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:45.300350 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" podStartSLOduration=1.3003358139999999 podStartE2EDuration="1.300335814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:45.298073898 +0000 UTC m=+2591.428487043" watchObservedRunningTime="2026-07-07 16:24:45.300335814 +0000 UTC m=+2591.430748960" Jul 07 16:24:45.972668 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:45.972644 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58f4b5668f-brlc2_91298c97-a9ef-4278-858c-5bf93c0a9694/console/0.log" Jul 07 16:24:47.885508 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:47.885479 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgwz7_b0bc8fd0-fce0-499d-a241-2df1853d1f60/dns/0.log" Jul 07 16:24:47.911428 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:47.911401 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgwz7_b0bc8fd0-fce0-499d-a241-2df1853d1f60/kube-rbac-proxy/0.log" Jul 07 16:24:48.040976 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:48.040950 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-psmv2_cd55f852-d386-432b-a490-583cc2a4db3d/dns-node-resolver/0.log" Jul 07 16:24:48.753843 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:48.753819 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vd7wn_185cd267-f8e8-4e94-a620-9b5ac374fe19/node-ca/0.log" Jul 07 16:24:49.934269 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:49.934239 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ccc59c597-hdv58_1f958698-a875-466d-af7f-3a4e6c6528eb/kube-auth-proxy/0.log" Jul 07 16:24:50.093467 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:50.093436 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-r7ldp_b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a/payload-pre-processing/0.log" Jul 07 16:24:50.143079 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:50.143054 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-ctm2s_cbc19cca-d26f-4954-ba6b-c524423efd2f/payload-processing/0.log" Jul 07 16:24:50.902836 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:50.902791 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-66js4_3d1ee745-d61d-4d6d-bda8-6eb2417bd150/serve-healthcheck-canary/0.log" Jul 07 16:24:51.296111 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:51.296034 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qslmw" Jul 07 16:24:51.598687 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:51.598659 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r7ldp_7aae2321-d8a2-4a29-a053-84d7d79b1f6a/kube-rbac-proxy/0.log" Jul 07 16:24:51.624674 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:51.624650 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r7ldp_7aae2321-d8a2-4a29-a053-84d7d79b1f6a/exporter/0.log" Jul 07 16:24:51.652025 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:51.652004 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r7ldp_7aae2321-d8a2-4a29-a053-84d7d79b1f6a/extractor/0.log" Jul 07 16:24:55.002566 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:55.002471 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685dff6fcf-ghv85_3c3503fe-bc75-4ace-a40c-af18bac782e9/manager/0.log" Jul 07 16:24:55.030433 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:55.030408 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685dff6fcf-ghv85_3c3503fe-bc75-4ace-a40c-af18bac782e9/copy-manifests/0.log" Jul 07 16:24:55.428453 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:55.428428 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ccc59c597-hdv58_1f958698-a875-466d-af7f-3a4e6c6528eb/kube-auth-proxy/0.log" Jul 07 16:24:55.530967 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:55.530902 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-r7ldp_b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a/payload-pre-processing/0.log" Jul 07 16:24:55.566836 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:55.566812 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-ctm2s_cbc19cca-d26f-4954-ba6b-c524423efd2f/payload-processing/0.log" Jul 07 16:24:56.769790 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:56.769706 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7444699877-kqrqp_417000df-04e6-4ca8-af71-6a678f95e4cb/manager/0.log" Jul 07 16:24:56.940412 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:56.940384 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ccc59c597-hdv58_1f958698-a875-466d-af7f-3a4e6c6528eb/kube-auth-proxy/0.log" Jul 07 16:24:57.043843 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:57.043754 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-r7ldp_b3302ffd-4765-4a5a-9ee7-87a7f4b0a47a/payload-pre-processing/0.log" Jul 07 16:24:57.082207 ip-10-0-128-121 kubenswrapper[2583]: I0707 16:24:57.082173 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-ctm2s_cbc19cca-d26f-4954-ba6b-c524423efd2f/payload-processing/0.log"