Jul 18 01:46:12.586507 ip-10-0-135-157 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 18 01:46:12.586521 ip-10-0-135-157 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 18 01:46:12.586554 ip-10-0-135-157 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 18 01:46:12.586867 ip-10-0-135-157 systemd[1]: Failed to start Kubernetes Kubelet. Jul 18 01:46:22.835646 ip-10-0-135-157 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 18 01:46:22.835662 ip-10-0-135-157 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4d9ba52e2825496eb1b0cb3262c97694 -- Jul 18 01:48:15.549036 ip-10-0-135-157 systemd[1]: Starting Kubernetes Kubelet... Jul 18 01:48:15.969616 ip-10-0-135-157 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 01:48:15.969616 ip-10-0-135-157 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 18 01:48:15.969616 ip-10-0-135-157 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 01:48:15.969616 ip-10-0-135-157 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 18 01:48:15.969616 ip-10-0-135-157 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 01:48:15.970590 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.970458 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 18 01:48:15.975821 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975805 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975823 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975827 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975830 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975833 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975837 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975840 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975842 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975845 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975848 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975851 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975853 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975856 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975858 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975861 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975864 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975867 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975870 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 01:48:15.975863 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975872 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975875 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975878 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975880 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975884 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975887 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975889 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975893 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975896 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975899 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975903 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975905 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975908 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975912 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975916 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975919 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975922 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975924 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975927 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 01:48:15.976289 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975930 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975932 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975935 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975937 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975940 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975942 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975944 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975947 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975949 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975951 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975954 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975958 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975960 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975963 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975965 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975969 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975971 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975974 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975976 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975979 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 01:48:15.976734 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975981 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975984 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975986 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975988 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975991 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975993 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975996 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.975998 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976000 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976003 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976005 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976007 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976010 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976013 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976015 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976018 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976020 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976023 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976026 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976029 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 01:48:15.977234 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976031 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976033 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976036 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976039 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976041 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976043 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976046 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976050 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976052 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976055 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976057 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976060 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976062 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976065 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976067 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976070 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976073 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976076 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976078 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976081 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 01:48:15.977722 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976083 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 01:48:15.978217 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.976086 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 01:48:15.979597 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979583 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 01:48:15.979597 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979596 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979600 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979603 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979606 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979608 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979611 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979614 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979616 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979618 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979621 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979629 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979632 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979635 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979638 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979640 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979642 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979645 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979647 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979650 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979652 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 01:48:15.979656 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979654 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979657 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979660 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979662 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979665 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979668 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979670 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979673 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979675 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979677 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979680 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979683 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979685 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979687 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979690 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979692 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979695 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979697 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979699 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979702 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 01:48:15.980194 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979704 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979708 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979712 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979716 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979718 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979721 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979724 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979726 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979729 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979731 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979733 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979750 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979753 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979755 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979760 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979764 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979767 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979771 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979774 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 01:48:15.980679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979777 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979779 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979782 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979784 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979787 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979789 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979792 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979794 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979798 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979800 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979803 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979805 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979808 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979810 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979813 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979815 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979818 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979821 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979824 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979826 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 01:48:15.981151 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979829 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979831 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979833 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979836 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979839 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979841 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979844 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979847 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979850 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979852 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979855 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979857 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979860 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979863 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979867 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979870 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979873 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979875 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.979878 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 01:48:15.981641 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980258 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980264 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980267 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980269 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980272 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980275 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980277 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980280 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980282 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980285 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980288 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980291 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980294 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980298 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980301 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980303 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980306 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980309 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980311 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 01:48:15.982110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980314 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980317 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980319 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980322 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980325 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980327 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980330 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980332 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980335 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980338 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980340 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980342 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980345 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980347 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980349 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980352 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980354 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980356 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980359 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980361 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 01:48:15.982569 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980363 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980366 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980368 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980370 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980373 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980376 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980378 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980381 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980383 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980386 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980389 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980391 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980394 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980396 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980399 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980402 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980404 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980406 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980408 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980411 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 01:48:15.983067 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980413 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980416 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980418 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980420 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980424 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980426 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980429 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980431 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980434 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980436 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980438 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980441 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980443 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980446 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980448 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980451 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980453 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980456 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980460 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 01:48:15.983531 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980463 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980466 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980469 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980472 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980475 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980477 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980480 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980482 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980485 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980487 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980491 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980494 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980498 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980500 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980503 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980505 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980508 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980510 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980512 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 01:48:15.983995 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980515 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.980517 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980594 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980602 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980608 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980612 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980616 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980619 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980623 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980627 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980631 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980633 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980637 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980641 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980644 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980647 2582 flags.go:64] FLAG: --cgroup-root="" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980650 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980653 2582 flags.go:64] FLAG: --client-ca-file="" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980656 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980659 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980664 2582 flags.go:64] FLAG: --cluster-domain="" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980667 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980670 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980673 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 18 01:48:15.984442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980676 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980680 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980683 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980686 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980689 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980692 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980695 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980698 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980701 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980704 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980708 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980711 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980713 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980716 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980719 2582 flags.go:64] FLAG: --enable-server="true" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980722 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980726 2582 flags.go:64] FLAG: --event-burst="100" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980729 2582 flags.go:64] FLAG: --event-qps="50" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980732 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980750 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980754 2582 flags.go:64] FLAG: --eviction-hard="" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980758 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980761 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980764 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980768 2582 flags.go:64] FLAG: --eviction-soft="" Jul 18 01:48:15.985151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980771 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980774 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980777 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980780 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980782 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980785 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980788 2582 flags.go:64] FLAG: --feature-gates="" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980792 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980795 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980798 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980801 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980803 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980806 2582 flags.go:64] FLAG: --help="false" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980810 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-135-157.ec2.internal" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980813 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980815 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980818 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980821 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980824 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980827 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980830 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980833 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980835 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 18 01:48:15.985750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980838 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980841 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980844 2582 flags.go:64] FLAG: --kube-reserved="" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980846 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980849 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980852 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980858 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980861 2582 flags.go:64] FLAG: --lock-file="" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980863 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980866 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980869 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980875 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980877 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980880 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980883 2582 flags.go:64] FLAG: --logging-format="text" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980886 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980889 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980892 2582 flags.go:64] FLAG: --manifest-url="" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980895 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980899 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980903 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980906 2582 flags.go:64] FLAG: --max-pods="110" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980909 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980912 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980915 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 18 01:48:15.986335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980917 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980920 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980923 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980927 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980934 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980937 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980940 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980943 2582 flags.go:64] FLAG: --pod-cidr="" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980946 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980951 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980954 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980957 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980960 2582 flags.go:64] FLAG: --port="10250" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980963 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980967 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a7bc05905fe33171" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980971 2582 flags.go:64] FLAG: --qos-reserved="" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980974 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980976 2582 flags.go:64] FLAG: --register-node="true" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980979 2582 flags.go:64] FLAG: --register-with-taints="" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980983 2582 flags.go:64] FLAG: --registry-burst="10" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980985 2582 flags.go:64] FLAG: --registry-qps="5" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980988 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980991 2582 flags.go:64] FLAG: --reserved-memory="" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980994 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.980997 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 18 01:48:15.986987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981000 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981003 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981005 2582 flags.go:64] FLAG: --runonce="false" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981008 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981011 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981013 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981016 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981019 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981022 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981027 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981030 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981033 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981035 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981038 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981041 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981044 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981046 2582 flags.go:64] FLAG: --system-cgroups="" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981049 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981054 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981057 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981059 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981066 2582 flags.go:64] FLAG: --tls-min-version="" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981069 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981072 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981075 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 18 01:48:15.987586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981077 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981080 2582 flags.go:64] FLAG: --v="2" Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981084 2582 flags.go:64] FLAG: --version="false" Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981088 2582 flags.go:64] FLAG: --vmodule="" Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981091 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.981094 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981209 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981213 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981216 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981218 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981221 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981223 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981226 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981228 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981230 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981233 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981238 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981240 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981243 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981245 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981248 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 01:48:15.988229 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981250 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981253 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981255 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981258 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981260 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981265 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981269 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981273 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981276 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981280 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981283 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981286 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981288 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981291 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981293 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981296 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981298 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981301 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981303 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 01:48:15.988733 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981306 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981308 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981311 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981313 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981315 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981318 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981320 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981323 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981327 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981329 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981331 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981334 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981337 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981339 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981341 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981344 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981346 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981348 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981351 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981353 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 01:48:15.989200 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981356 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981359 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981361 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981372 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981375 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981377 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981379 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981382 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981384 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981386 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981390 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981394 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981396 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981399 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981402 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981404 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981407 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981409 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981411 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 01:48:15.989724 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981415 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981422 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981425 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981427 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981430 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981432 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981435 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981437 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981440 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981442 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981445 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981448 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981451 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981456 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981459 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981461 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981463 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981471 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981474 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981476 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 01:48:15.990196 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981479 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 01:48:15.990679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981481 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 01:48:15.990679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981484 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 01:48:15.990679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981486 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 01:48:15.990679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981488 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 01:48:15.990679 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.981491 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 01:48:15.990679 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.982048 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 01:48:15.991032 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.991013 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 18 01:48:15.991065 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.991032 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 18 01:48:15.991065 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.991049 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 18 01:48:15.991065 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.991056 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991114 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991118 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991122 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991125 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991128 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991130 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991133 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991136 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991138 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991141 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 01:48:15.991140 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991144 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991147 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991149 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991152 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991154 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991157 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991159 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991162 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991164 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991167 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991169 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991171 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991174 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991177 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991180 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991184 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991187 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991190 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991193 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 01:48:15.991409 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991196 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991199 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991202 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991205 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991207 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991210 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991212 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991215 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991218 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991220 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991223 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991226 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991229 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991233 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991235 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991239 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991242 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991245 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991248 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 01:48:15.991905 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991251 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991254 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991257 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991259 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991262 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991265 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991267 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991269 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991272 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991274 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991277 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991279 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991281 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991284 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991286 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991289 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991292 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991294 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991296 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991299 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 01:48:15.992351 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991301 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991304 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991306 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991308 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991310 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991313 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991316 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991318 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991321 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991324 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991326 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991329 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991331 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991334 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991336 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991338 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991341 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991343 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991346 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 01:48:15.992858 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991348 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991350 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991353 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991355 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991358 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991360 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991362 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991365 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991368 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991370 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991373 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991376 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.991382 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991497 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 01:48:15.993305 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991502 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991505 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991509 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991511 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991514 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991516 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991519 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991522 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991525 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991528 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991530 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991533 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991535 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991539 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991542 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991544 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991547 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991549 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991551 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 01:48:15.993649 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991554 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991556 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991559 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991561 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991564 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991566 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991568 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991571 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991574 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991576 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991579 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991581 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991584 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991586 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991588 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991591 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991593 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991596 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991598 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991601 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 01:48:15.994163 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991603 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991606 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991608 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991611 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991613 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991616 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991618 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991621 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991623 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991627 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991630 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991633 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991636 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991639 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991641 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991644 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991647 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991649 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991652 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 01:48:15.994639 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991654 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991657 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991660 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991662 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991664 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991667 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991669 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991672 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991674 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991676 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991679 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991681 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991684 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991687 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991689 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991692 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991694 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991697 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991699 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 01:48:15.995125 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991701 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991704 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991706 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991709 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991711 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991714 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991717 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991720 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991723 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991726 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991728 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991731 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991734 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991754 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991757 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991759 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991763 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991766 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991768 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991771 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 01:48:15.995594 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:15.991773 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 18 01:48:15.996081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.991779 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 01:48:15.996081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.991892 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 18 01:48:15.996081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.993920 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 18 01:48:15.996081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.995334 2582 server.go:1019] "Starting client certificate rotation" Jul 18 01:48:15.996081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.995432 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 01:48:15.996081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.995468 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 01:48:15.998095 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:15.998080 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 01:48:16.002792 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.002768 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 01:48:16.013145 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.013124 2582 log.go:25] "Validated CRI v1 runtime API" Jul 18 01:48:16.016065 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.016051 2582 log.go:25] "Validated CRI v1 image API" Jul 18 01:48:16.016787 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.016770 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 18 01:48:16.019489 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.019471 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 01:48:16.019957 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.019928 2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9d3589a4-4e4c-4b4c-8054-fe985b568305:/dev/nvme0n1p3 cf7016f6-3d7d-42bf-9b50-a6f85d09015e:/dev/nvme0n1p4] Jul 18 01:48:16.020004 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.019959 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 18 01:48:16.024915 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.024806 2582 manager.go:216] Machine: {Timestamp:2026-07-18 01:48:16.023491168 +0000 UTC m=+0.354732141 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103385 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c7ea5c3fe032d1c6df513fc4a924f SystemUUID:ec2c7ea5-c3fe-032d-1c6d-f513fc4a924f BootID:4d9ba52e-2825-496e-b1b0-cb3262c97694 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f6:26:ab:2f:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f6:26:ab:2f:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:c9:22:4b:d1:44 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 18 01:48:16.024915 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.024908 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 18 01:48:16.025036 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.024979 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 18 01:48:16.028179 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.028152 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 18 01:48:16.028319 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.028181 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-157.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 18 01:48:16.029047 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.029037 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 18 01:48:16.029078 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.029049 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 18 01:48:16.029078 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.029056 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 01:48:16.030007 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.029996 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 01:48:16.031239 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.031230 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 18 01:48:16.034540 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.034529 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 18 01:48:16.034660 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.034651 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 18 01:48:16.034689 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.034681 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 18 01:48:16.034722 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.034694 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 18 01:48:16.034722 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.034716 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 18 01:48:16.034790 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.034730 2582 kubelet.go:398] "Adding apiserver pod source" Jul 18 01:48:16.034790 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.034753 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 18 01:48:16.035109 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.035089 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 01:48:16.035109 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.035107 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 01:48:16.037151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.037133 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 18 01:48:16.038926 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.038912 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 18 01:48:16.038974 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.038943 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 18 01:48:16.040131 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040119 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 18 01:48:16.040179 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040136 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 18 01:48:16.040179 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040141 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 18 01:48:16.040179 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040147 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 18 01:48:16.040179 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040156 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 18 01:48:16.040179 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040162 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 18 01:48:16.040179 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040169 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 18 01:48:16.040179 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040174 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 18 01:48:16.040179 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040182 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 18 01:48:16.040383 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040187 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 18 01:48:16.040383 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040195 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 18 01:48:16.040383 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.040201 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 18 01:48:16.041261 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.041250 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 18 01:48:16.041261 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.041260 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 18 01:48:16.041901 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.041880 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8dpd8" Jul 18 01:48:16.043901 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.043874 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 18 01:48:16.043901 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.043867 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 18 01:48:16.044022 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.043916 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-157.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 18 01:48:16.044473 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.044461 2582 server.go:1268] "Started kubelet" Jul 18 01:48:16.046432 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.044539 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 18 01:48:16.046582 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.046561 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 18 01:48:16.046873 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.046409 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 18 01:48:16.047114 ip-10-0-135-157 systemd[1]: Started Kubernetes Kubelet. Jul 18 01:48:16.048356 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.048340 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 18 01:48:16.048926 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.048910 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 18 01:48:16.050067 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.050030 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8dpd8" Jul 18 01:48:16.052909 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.052886 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 18 01:48:16.053113 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.052174 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-157.ec2.internal.18c33f22cb7fbce5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-157.ec2.internal,UID:ip-10-0-135-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-157.ec2.internal,},FirstTimestamp:2026-07-18 01:48:16.044432613 +0000 UTC m=+0.375673640,LastTimestamp:2026-07-18 01:48:16.044432613 +0000 UTC m=+0.375673640,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-157.ec2.internal,}" Jul 18 01:48:16.056513 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.056496 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 18 01:48:16.056592 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.056514 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 18 01:48:16.056655 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.056644 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.057256 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.057239 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 18 01:48:16.057256 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.057244 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 18 01:48:16.057387 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.057267 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 18 01:48:16.057443 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.057398 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 18 01:48:16.057443 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.057409 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 18 01:48:16.057538 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.057443 2582 factory.go:55] Registering systemd factory Jul 18 01:48:16.057538 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.057458 2582 factory.go:223] Registration of the systemd container factory successfully Jul 18 01:48:16.059817 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.059802 2582 factory.go:153] Registering CRI-O factory Jul 18 01:48:16.059905 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.059819 2582 factory.go:223] Registration of the crio container factory successfully Jul 18 01:48:16.059905 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.059872 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 18 01:48:16.059905 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.059901 2582 factory.go:103] Registering Raw factory Jul 18 01:48:16.060050 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.059916 2582 manager.go:1196] Started watching for new ooms in manager Jul 18 01:48:16.060434 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.060404 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 01:48:16.060551 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.060539 2582 manager.go:349] Starting recovery of all containers Jul 18 01:48:16.063778 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.063754 2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-157.ec2.internal\" not found" node="ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.070431 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.070326 2582 manager.go:354] Recovery completed Jul 18 01:48:16.074191 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.074180 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 01:48:16.075874 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.075858 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientMemory" Jul 18 01:48:16.075931 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.075888 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 01:48:16.075931 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.075898 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientPID" Jul 18 01:48:16.076383 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.076369 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 18 01:48:16.076383 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.076379 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 01:48:16.076383 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.076382 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 18 01:48:16.076533 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.076402 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 18 01:48:16.077066 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.077051 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientMemory" Jul 18 01:48:16.077129 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.077079 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 01:48:16.077129 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.077092 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientPID" Jul 18 01:48:16.077702 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.077689 2582 policy_none.go:49] "None policy: Start" Jul 18 01:48:16.077777 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.077708 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 18 01:48:16.077777 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.077722 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 18 01:48:16.078955 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.078943 2582 policy_none.go:47] "Start" Jul 18 01:48:16.128355 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.128335 2582 manager.go:337] "Starting Device Plugin manager" Jul 18 01:48:16.128436 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.128371 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 18 01:48:16.128436 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.128383 2582 server.go:87] "Starting device plugin registration server" Jul 18 01:48:16.128668 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.128655 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 18 01:48:16.128719 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.128674 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 18 01:48:16.128829 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.128814 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 18 01:48:16.128901 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.128884 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 18 01:48:16.128901 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.128893 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 18 01:48:16.129436 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.129418 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 18 01:48:16.129520 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.129468 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.186861 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.186830 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 18 01:48:16.188184 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.188155 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 18 01:48:16.188275 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.188203 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 18 01:48:16.188275 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.188232 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 18 01:48:16.188352 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.188272 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 18 01:48:16.191005 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.190981 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 01:48:16.229308 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.229218 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 01:48:16.230203 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.230183 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientMemory" Jul 18 01:48:16.230311 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.230216 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 01:48:16.230311 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.230228 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientPID" Jul 18 01:48:16.230311 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.230257 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.239703 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.239679 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.239833 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.239709 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-157.ec2.internal\": node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.259096 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.259073 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.289093 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.288460 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal"] Jul 18 01:48:16.289196 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.289137 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 01:48:16.290756 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.290714 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientMemory" Jul 18 01:48:16.290863 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.290765 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 01:48:16.290863 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.290776 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientPID" Jul 18 01:48:16.292006 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.291994 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 01:48:16.292127 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.292110 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.292163 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.292145 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 01:48:16.292820 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.292804 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientMemory" Jul 18 01:48:16.292889 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.292804 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientMemory" Jul 18 01:48:16.292889 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.292860 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 01:48:16.292889 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.292872 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientPID" Jul 18 01:48:16.292889 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.292835 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 01:48:16.293002 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.292900 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientPID" Jul 18 01:48:16.294116 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.294100 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.294190 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.294125 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 01:48:16.294831 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.294814 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientMemory" Jul 18 01:48:16.294929 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.294841 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 01:48:16.294929 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.294855 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeHasSufficientPID" Jul 18 01:48:16.323506 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.323482 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-157.ec2.internal\" not found" node="ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.326928 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.326911 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-157.ec2.internal\" not found" node="ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.359575 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.359554 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.359697 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.359606 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cdf25076591eab9f29e707e0b8f3f259-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal\" (UID: \"cdf25076591eab9f29e707e0b8f3f259\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.359697 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.359631 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdf25076591eab9f29e707e0b8f3f259-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal\" (UID: \"cdf25076591eab9f29e707e0b8f3f259\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.359697 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.359652 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38b08105c0224d562f53bf7954a3df13-config\") pod \"kube-apiserver-proxy-ip-10-0-135-157.ec2.internal\" (UID: \"38b08105c0224d562f53bf7954a3df13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.459891 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.459851 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.460046 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.459900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cdf25076591eab9f29e707e0b8f3f259-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal\" (UID: \"cdf25076591eab9f29e707e0b8f3f259\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.460046 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.459928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdf25076591eab9f29e707e0b8f3f259-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal\" (UID: \"cdf25076591eab9f29e707e0b8f3f259\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.460046 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.459947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38b08105c0224d562f53bf7954a3df13-config\") pod \"kube-apiserver-proxy-ip-10-0-135-157.ec2.internal\" (UID: \"38b08105c0224d562f53bf7954a3df13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.460046 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.460002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38b08105c0224d562f53bf7954a3df13-config\") pod \"kube-apiserver-proxy-ip-10-0-135-157.ec2.internal\" (UID: \"38b08105c0224d562f53bf7954a3df13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.460046 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.460024 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdf25076591eab9f29e707e0b8f3f259-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal\" (UID: \"cdf25076591eab9f29e707e0b8f3f259\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.460046 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.460002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cdf25076591eab9f29e707e0b8f3f259-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal\" (UID: \"cdf25076591eab9f29e707e0b8f3f259\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.560528 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.560459 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.626702 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.626673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.630707 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.630562 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal" Jul 18 01:48:16.661380 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.661349 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.761802 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.761728 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.862217 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.862136 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.962911 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:16.962885 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:16.995173 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.995113 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 18 01:48:16.995837 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.995280 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 01:48:16.995837 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:16.995328 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 01:48:17.055659 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.055621 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-17 01:43:16 +0000 UTC" deadline="2028-02-04 14:37:12.534097224 +0000 UTC" Jul 18 01:48:17.055659 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.055652 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13596h48m55.478448132s" Jul 18 01:48:17.064044 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:17.064001 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:17.132642 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:17.132592 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b08105c0224d562f53bf7954a3df13.slice/crio-f4253bdcf822ed87da8eb6e4cec2bad2261c023bac5592bbd49e1037185e9bf1 WatchSource:0}: Error finding container f4253bdcf822ed87da8eb6e4cec2bad2261c023bac5592bbd49e1037185e9bf1: Status 404 returned error can't find the container with id f4253bdcf822ed87da8eb6e4cec2bad2261c023bac5592bbd49e1037185e9bf1 Jul 18 01:48:17.132859 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:17.132836 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf25076591eab9f29e707e0b8f3f259.slice/crio-d9de7a33c5270e4301c648007720483e91ef94f1ccd592dfd1583613c389909d WatchSource:0}: Error finding container d9de7a33c5270e4301c648007720483e91ef94f1ccd592dfd1583613c389909d: Status 404 returned error can't find the container with id d9de7a33c5270e4301c648007720483e91ef94f1ccd592dfd1583613c389909d Jul 18 01:48:17.137727 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.137709 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 01:48:17.164857 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:17.164823 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-157.ec2.internal\" not found" Jul 18 01:48:17.191441 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.191411 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 01:48:17.192409 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.192362 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal" event={"ID":"38b08105c0224d562f53bf7954a3df13","Type":"ContainerStarted","Data":"f4253bdcf822ed87da8eb6e4cec2bad2261c023bac5592bbd49e1037185e9bf1"} Jul 18 01:48:17.193468 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.193441 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" event={"ID":"cdf25076591eab9f29e707e0b8f3f259","Type":"ContainerStarted","Data":"d9de7a33c5270e4301c648007720483e91ef94f1ccd592dfd1583613c389909d"} Jul 18 01:48:17.261107 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.261067 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" Jul 18 01:48:17.273408 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.273384 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 01:48:17.275401 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.275386 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal" Jul 18 01:48:17.282610 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.282594 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 01:48:17.310389 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.310363 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 01:48:17.873633 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:17.873603 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 01:48:18.035977 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.035927 2582 apiserver.go:52] "Watching apiserver" Jul 18 01:48:18.042536 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.042509 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 18 01:48:18.042967 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.042935 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ndt4g","kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z","openshift-dns/node-resolver-22pm6","openshift-image-registry/node-ca-ghg58","openshift-multus/multus-49dkv","openshift-multus/network-metrics-daemon-v4r5t","openshift-network-diagnostics/network-check-target-cxhwx","kube-system/konnectivity-agent-n6gnr","openshift-cluster-node-tuning-operator/tuned-6ksp2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal","openshift-multus/multus-additional-cni-plugins-lwdx9","openshift-network-operator/iptables-alerter-2r886","openshift-ovn-kubernetes/ovnkube-node-6vl2w"] Jul 18 01:48:18.044897 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.044875 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.046693 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.046673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.047475 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.047327 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qjmhb\"" Jul 18 01:48:18.047475 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.047334 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 18 01:48:18.047475 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.047376 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 18 01:48:18.048044 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.048023 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.048927 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.048907 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 18 01:48:18.049025 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.049006 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 18 01:48:18.049025 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.049008 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 18 01:48:18.049302 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.049285 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8hnd4\"" Jul 18 01:48:18.049479 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.049461 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.050203 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.050106 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 18 01:48:18.050393 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.050359 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 18 01:48:18.050670 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.050649 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rx8t9\"" Jul 18 01:48:18.050982 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.050960 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 18 01:48:18.051085 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.050976 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.052025 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.052008 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 18 01:48:18.052120 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.052057 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 18 01:48:18.052185 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.052135 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 18 01:48:18.052241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.052195 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 18 01:48:18.052296 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.052253 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k64gx\"" Jul 18 01:48:18.052387 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.052367 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 18 01:48:18.052484 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.052468 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.053397 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.053166 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 18 01:48:18.053397 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.053250 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 18 01:48:18.053397 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.053250 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 18 01:48:18.053567 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.053550 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-426dm\"" Jul 18 01:48:18.055131 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.055042 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 18 01:48:18.055131 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.055095 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6rmd5\"" Jul 18 01:48:18.055582 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.055560 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.056871 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.056852 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 18 01:48:18.057092 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.057066 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:18.057212 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.057158 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:18.058233 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.057880 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l7p4d\"" Jul 18 01:48:18.058328 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.058231 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 18 01:48:18.058763 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.058747 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 18 01:48:18.061655 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.059437 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:18.062011 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.061987 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 18 01:48:18.062190 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.062173 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5k774\"" Jul 18 01:48:18.062387 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.062367 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 18 01:48:18.062640 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.062518 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:18.062773 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.062616 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:18.064692 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.064671 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.064807 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.064767 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:18.066330 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.066311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.069529 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.069497 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 18 01:48:18.069774 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.069737 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 18 01:48:18.069852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.069820 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-22s56\"" Jul 18 01:48:18.070085 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.070068 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 18 01:48:18.070136 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.070088 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 18 01:48:18.070188 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.070167 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 18 01:48:18.070237 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.070189 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 18 01:48:18.071480 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071457 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4d10c59-9d5a-476b-8a8f-091d79787977-hosts-file\") pod \"node-resolver-22pm6\" (UID: \"b4d10c59-9d5a-476b-8a8f-091d79787977\") " pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.071582 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-systemd\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.071582 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071519 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-var-lib-kubelet\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.071582 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071543 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-tuned\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.071582 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071567 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8-host\") pod \"node-ca-ghg58\" (UID: \"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8\") " pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.071826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071591 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-cni-binary-copy\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.071826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-var-lib-cni-bin\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.071826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-socket-dir-parent\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.071826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-run-netns\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.071826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071699 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-run-multus-certs\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.071826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071724 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f223dcd3-22b0-4b2e-aad8-cc9d66639fd1-iptables-alerter-script\") pod \"iptables-alerter-2r886\" (UID: \"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1\") " pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.071826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071750 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-cnibin\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.071826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-conf-dir\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.071826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-os-release\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/440cebb3-367e-4f50-9418-46b7182e7520-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071894 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4d10c59-9d5a-476b-8a8f-091d79787977-tmp-dir\") pod \"node-resolver-22pm6\" (UID: \"b4d10c59-9d5a-476b-8a8f-091d79787977\") " pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071931 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwwk\" (UniqueName: \"kubernetes.io/projected/f223dcd3-22b0-4b2e-aad8-cc9d66639fd1-kube-api-access-svwwk\") pod \"iptables-alerter-2r886\" (UID: \"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1\") " pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.071982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-hostroot\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072009 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-kubernetes\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072033 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrfrw\" (UniqueName: \"kubernetes.io/projected/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-kube-api-access-mrfrw\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072057 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgpk\" (UniqueName: \"kubernetes.io/projected/b4d10c59-9d5a-476b-8a8f-091d79787977-kube-api-access-bwgpk\") pod \"node-resolver-22pm6\" (UID: \"b4d10c59-9d5a-476b-8a8f-091d79787977\") " pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072081 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-modprobe-d\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072105 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-sysconfig\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072129 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-lib-modules\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-sys-fs\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072189 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndxc\" (UniqueName: \"kubernetes.io/projected/3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8-kube-api-access-zndxc\") pod \"node-ca-ghg58\" (UID: \"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8\") " pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-run-k8s-cni-cncf-io\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.072241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-sysctl-conf\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072256 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-sys\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072271 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-host\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f223dcd3-22b0-4b2e-aad8-cc9d66639fd1-host-slash\") pod \"iptables-alerter-2r886\" (UID: \"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1\") " pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072308 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-socket-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-var-lib-kubelet\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072345 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-system-cni-dir\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-device-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072376 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/440cebb3-367e-4f50-9418-46b7182e7520-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072407 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072423 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-registration-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-cni-dir\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-var-lib-cni-multus\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072502 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-daemon-config\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072539 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-etc-kubernetes\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072564 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-sysctl-d\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.072997 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-cnibin\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072650 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f75d2da2-02ba-40e4-8a9d-efe762a83e51-agent-certs\") pod \"konnectivity-agent-n6gnr\" (UID: \"f75d2da2-02ba-40e4-8a9d-efe762a83e51\") " pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k5hb\" (UniqueName: \"kubernetes.io/projected/4d3eb90f-655a-4e6e-acca-0467391032f1-kube-api-access-7k5hb\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9fk\" (UniqueName: \"kubernetes.io/projected/3d6aef8f-9538-4ad9-8ce3-9463040b5054-kube-api-access-ds9fk\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f75d2da2-02ba-40e4-8a9d-efe762a83e51-konnectivity-ca\") pod \"konnectivity-agent-n6gnr\" (UID: \"f75d2da2-02ba-40e4-8a9d-efe762a83e51\") " pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6aef8f-9538-4ad9-8ce3-9463040b5054-tmp\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brd45\" (UniqueName: \"kubernetes.io/projected/440cebb3-367e-4f50-9418-46b7182e7520-kube-api-access-brd45\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072918 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8-serviceca\") pod \"node-ca-ghg58\" (UID: \"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8\") " pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-os-release\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072966 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/440cebb3-367e-4f50-9418-46b7182e7520-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.072982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-run\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.073014 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.073770 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.073038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-system-cni-dir\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.078482 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.078455 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 01:48:18.094218 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.094194 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p7gts" Jul 18 01:48:18.103138 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.103116 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p7gts" Jul 18 01:48:18.158969 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.158887 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 18 01:48:18.173847 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.173811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-os-release\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.174113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.173860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/440cebb3-367e-4f50-9418-46b7182e7520-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.174113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.173938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4d10c59-9d5a-476b-8a8f-091d79787977-tmp-dir\") pod \"node-resolver-22pm6\" (UID: \"b4d10c59-9d5a-476b-8a8f-091d79787977\") " pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.174113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.173970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwwk\" (UniqueName: \"kubernetes.io/projected/f223dcd3-22b0-4b2e-aad8-cc9d66639fd1-kube-api-access-svwwk\") pod \"iptables-alerter-2r886\" (UID: \"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1\") " pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.174113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.173968 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-os-release\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.174113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.173992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.174113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174020 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-hostroot\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.174113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-kubernetes\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174093 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-run-ovn\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.174476 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-cni-netd\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.174476 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174182 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-hostroot\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.174476 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174210 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.174476 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174235 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-kubernetes\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174476 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174146 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrfrw\" (UniqueName: \"kubernetes.io/projected/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-kube-api-access-mrfrw\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.174476 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174302 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgpk\" (UniqueName: \"kubernetes.io/projected/b4d10c59-9d5a-476b-8a8f-091d79787977-kube-api-access-bwgpk\") pod \"node-resolver-22pm6\" (UID: \"b4d10c59-9d5a-476b-8a8f-091d79787977\") " pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.174476 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-modprobe-d\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174476 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-sysconfig\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174476 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174415 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4d10c59-9d5a-476b-8a8f-091d79787977-tmp-dir\") pod \"node-resolver-22pm6\" (UID: \"b4d10c59-9d5a-476b-8a8f-091d79787977\") " pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-modprobe-d\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174517 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/440cebb3-367e-4f50-9418-46b7182e7520-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174542 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-sysconfig\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-lib-modules\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-sys-fs\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndxc\" (UniqueName: \"kubernetes.io/projected/3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8-kube-api-access-zndxc\") pod \"node-ca-ghg58\" (UID: \"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8\") " pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-run-k8s-cni-cncf-io\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174674 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-sys-fs\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174686 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-sysctl-conf\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174667 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-lib-modules\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-sys\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-host\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174774 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-run-k8s-cni-cncf-io\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174795 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f223dcd3-22b0-4b2e-aad8-cc9d66639fd1-host-slash\") pod \"iptables-alerter-2r886\" (UID: \"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1\") " pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174799 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-sys\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174826 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-sysctl-conf\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.174890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-run-openvswitch\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-host\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-socket-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174870 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f223dcd3-22b0-4b2e-aad8-cc9d66639fd1-host-slash\") pod \"iptables-alerter-2r886\" (UID: \"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1\") " pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174893 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-var-lib-kubelet\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174916 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-var-lib-kubelet\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174932 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-log-socket\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174974 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72f650cd-84a0-41e8-b71e-982622024be0-ovnkube-script-lib\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.174976 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-socket-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175007 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqfr\" (UniqueName: \"kubernetes.io/projected/72f650cd-84a0-41e8-b71e-982622024be0-kube-api-access-cfqfr\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175037 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-system-cni-dir\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175071 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-device-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175088 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/440cebb3-367e-4f50-9418-46b7182e7520-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-slash\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175144 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-system-cni-dir\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175138 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-device-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.175663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-registration-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-cni-dir\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175224 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-registration-dir\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-var-lib-cni-multus\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175275 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-var-lib-cni-multus\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-daemon-config\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-cni-dir\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-etc-kubernetes\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-sysctl-d\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-cnibin\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-etc-kubernetes\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175405 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f75d2da2-02ba-40e4-8a9d-efe762a83e51-agent-certs\") pod \"konnectivity-agent-n6gnr\" (UID: \"f75d2da2-02ba-40e4-8a9d-efe762a83e51\") " pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175432 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-cnibin\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-cni-bin\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72f650cd-84a0-41e8-b71e-982622024be0-ovnkube-config\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.176526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175472 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-sysctl-d\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72f650cd-84a0-41e8-b71e-982622024be0-ovn-node-metrics-cert\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k5hb\" (UniqueName: \"kubernetes.io/projected/4d3eb90f-655a-4e6e-acca-0467391032f1-kube-api-access-7k5hb\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175627 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/440cebb3-367e-4f50-9418-46b7182e7520-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9fk\" (UniqueName: \"kubernetes.io/projected/3d6aef8f-9538-4ad9-8ce3-9463040b5054-kube-api-access-ds9fk\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175689 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f75d2da2-02ba-40e4-8a9d-efe762a83e51-konnectivity-ca\") pod \"konnectivity-agent-n6gnr\" (UID: \"f75d2da2-02ba-40e4-8a9d-efe762a83e51\") " pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-kubelet\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175725 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/440cebb3-367e-4f50-9418-46b7182e7520-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175754 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8gd\" (UniqueName: \"kubernetes.io/projected/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-kube-api-access-5z8gd\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6aef8f-9538-4ad9-8ce3-9463040b5054-tmp\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brd45\" (UniqueName: \"kubernetes.io/projected/440cebb3-367e-4f50-9418-46b7182e7520-kube-api-access-brd45\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8-serviceca\") pod \"node-ca-ghg58\" (UID: \"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8\") " pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-os-release\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175931 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55d05391-7c5e-416f-a050-ed1b32053280-dbus\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175958 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-systemd-units\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.175983 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-etc-openvswitch\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.177556 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176012 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-daemon-config\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176096 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/440cebb3-367e-4f50-9418-46b7182e7520-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-run\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-run-netns\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-run-systemd\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176209 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-node-log\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176231 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-run\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176179 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-os-release\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176271 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72f650cd-84a0-41e8-b71e-982622024be0-env-overrides\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176299 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-system-cni-dir\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4d10c59-9d5a-476b-8a8f-091d79787977-hosts-file\") pod \"node-resolver-22pm6\" (UID: \"b4d10c59-9d5a-476b-8a8f-091d79787977\") " pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-systemd\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176401 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f75d2da2-02ba-40e4-8a9d-efe762a83e51-konnectivity-ca\") pod \"konnectivity-agent-n6gnr\" (UID: \"f75d2da2-02ba-40e4-8a9d-efe762a83e51\") " pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176408 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-var-lib-kubelet\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176433 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d3eb90f-655a-4e6e-acca-0467391032f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.178407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176459 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4d10c59-9d5a-476b-8a8f-091d79787977-hosts-file\") pod \"node-resolver-22pm6\" (UID: \"b4d10c59-9d5a-476b-8a8f-091d79787977\") " pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-tuned\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176452 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8-serviceca\") pod \"node-ca-ghg58\" (UID: \"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8\") " pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176498 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-systemd\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55d05391-7c5e-416f-a050-ed1b32053280-kubelet-config\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d6aef8f-9538-4ad9-8ce3-9463040b5054-var-lib-kubelet\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176543 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-system-cni-dir\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176565 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-var-lib-openvswitch\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8-host\") pod \"node-ca-ghg58\" (UID: \"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8\") " pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-cni-binary-copy\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176628 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8-host\") pod \"node-ca-ghg58\" (UID: \"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8\") " pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176640 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-var-lib-cni-bin\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176654 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/440cebb3-367e-4f50-9418-46b7182e7520-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176667 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176695 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176708 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-var-lib-cni-bin\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-socket-dir-parent\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176778 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-run-netns\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176831 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-run-netns\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176858 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-socket-dir-parent\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-run-multus-certs\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.176988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f223dcd3-22b0-4b2e-aad8-cc9d66639fd1-iptables-alerter-script\") pod \"iptables-alerter-2r886\" (UID: \"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1\") " pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.177020 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-cnibin\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.177039 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-host-run-multus-certs\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.177043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-conf-dir\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.177078 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-multus-conf-dir\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.177148 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-cnibin\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.177300 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-cni-binary-copy\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.177489 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f223dcd3-22b0-4b2e-aad8-cc9d66639fd1-iptables-alerter-script\") pod \"iptables-alerter-2r886\" (UID: \"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1\") " pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.179456 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d6aef8f-9538-4ad9-8ce3-9463040b5054-etc-tuned\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.179528 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6aef8f-9538-4ad9-8ce3-9463040b5054-tmp\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.179948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.179788 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f75d2da2-02ba-40e4-8a9d-efe762a83e51-agent-certs\") pod \"konnectivity-agent-n6gnr\" (UID: \"f75d2da2-02ba-40e4-8a9d-efe762a83e51\") " pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:18.185179 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.185141 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 01:48:18.185309 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.185201 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 01:48:18.185309 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.185217 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6r7fn for pod openshift-network-diagnostics/network-check-target-cxhwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:18.185429 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.185326 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn podName:403160fb-538f-437d-ba5a-37ae0c7c89d9 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:18.685301984 +0000 UTC m=+3.016542961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6r7fn" (UniqueName: "kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn") pod "network-check-target-cxhwx" (UID: "403160fb-538f-437d-ba5a-37ae0c7c89d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:18.186394 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.186365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndxc\" (UniqueName: \"kubernetes.io/projected/3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8-kube-api-access-zndxc\") pod \"node-ca-ghg58\" (UID: \"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8\") " pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.186637 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.186569 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgpk\" (UniqueName: \"kubernetes.io/projected/b4d10c59-9d5a-476b-8a8f-091d79787977-kube-api-access-bwgpk\") pod \"node-resolver-22pm6\" (UID: \"b4d10c59-9d5a-476b-8a8f-091d79787977\") " pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.187946 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.187920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwwk\" (UniqueName: \"kubernetes.io/projected/f223dcd3-22b0-4b2e-aad8-cc9d66639fd1-kube-api-access-svwwk\") pod \"iptables-alerter-2r886\" (UID: \"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1\") " pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.188051 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.188002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9fk\" (UniqueName: \"kubernetes.io/projected/3d6aef8f-9538-4ad9-8ce3-9463040b5054-kube-api-access-ds9fk\") pod \"tuned-6ksp2\" (UID: \"3d6aef8f-9538-4ad9-8ce3-9463040b5054\") " pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.188215 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.188190 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brd45\" (UniqueName: \"kubernetes.io/projected/440cebb3-367e-4f50-9418-46b7182e7520-kube-api-access-brd45\") pod \"multus-additional-cni-plugins-lwdx9\" (UID: \"440cebb3-367e-4f50-9418-46b7182e7520\") " pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.188606 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.188581 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrfrw\" (UniqueName: \"kubernetes.io/projected/2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0-kube-api-access-mrfrw\") pod \"multus-49dkv\" (UID: \"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0\") " pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.189070 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.189051 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k5hb\" (UniqueName: \"kubernetes.io/projected/4d3eb90f-655a-4e6e-acca-0467391032f1-kube-api-access-7k5hb\") pod \"aws-ebs-csi-driver-node-csr8z\" (UID: \"4d3eb90f-655a-4e6e-acca-0467391032f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.277709 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277675 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.277709 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-cni-bin\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72f650cd-84a0-41e8-b71e-982622024be0-ovnkube-config\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72f650cd-84a0-41e8-b71e-982622024be0-ovn-node-metrics-cert\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277794 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-kubelet\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-cni-bin\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.277864 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-kubelet\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8gd\" (UniqueName: \"kubernetes.io/projected/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-kube-api-access-5z8gd\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.277941 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret podName:55d05391-7c5e-416f-a050-ed1b32053280 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:18.777920508 +0000 UTC m=+3.109161473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret") pod "global-pull-secret-syncer-ndt4g" (UID: "55d05391-7c5e-416f-a050-ed1b32053280") : object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277964 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55d05391-7c5e-416f-a050-ed1b32053280-dbus\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.277983 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.277987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-systemd-units\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278030 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-systemd-units\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-etc-openvswitch\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278099 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-run-netns\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-run-systemd\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278143 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55d05391-7c5e-416f-a050-ed1b32053280-dbus\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-node-log\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72f650cd-84a0-41e8-b71e-982622024be0-env-overrides\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-node-log\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55d05391-7c5e-416f-a050-ed1b32053280-kubelet-config\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-var-lib-openvswitch\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278263 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278304 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-run-netns\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-run-ovn\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.278481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278347 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-run-systemd\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278388 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-var-lib-openvswitch\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278391 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-cni-netd\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278401 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72f650cd-84a0-41e8-b71e-982622024be0-ovnkube-config\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278231 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-etc-openvswitch\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-cni-netd\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278428 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-run-openvswitch\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-log-socket\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278506 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55d05391-7c5e-416f-a050-ed1b32053280-kubelet-config\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.278517 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72f650cd-84a0-41e8-b71e-982622024be0-ovnkube-script-lib\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278544 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-log-socket\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278584 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-run-openvswitch\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278594 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqfr\" (UniqueName: \"kubernetes.io/projected/72f650cd-84a0-41e8-b71e-982622024be0-kube-api-access-cfqfr\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-run-ovn\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.278612 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs podName:6d8e552e-32ad-4c02-8e89-bc9d00d093e5 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:18.778595837 +0000 UTC m=+3.109836822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs") pod "network-metrics-daemon-v4r5t" (UID: "6d8e552e-32ad-4c02-8e89-bc9d00d093e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:18.279516 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-slash\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.280392 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278752 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72f650cd-84a0-41e8-b71e-982622024be0-env-overrides\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.280392 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.278748 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72f650cd-84a0-41e8-b71e-982622024be0-host-slash\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.280392 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.279067 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72f650cd-84a0-41e8-b71e-982622024be0-ovnkube-script-lib\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.281055 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.281030 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72f650cd-84a0-41e8-b71e-982622024be0-ovn-node-metrics-cert\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.286955 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.286930 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8gd\" (UniqueName: \"kubernetes.io/projected/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-kube-api-access-5z8gd\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:18.287092 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.286969 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqfr\" (UniqueName: \"kubernetes.io/projected/72f650cd-84a0-41e8-b71e-982622024be0-kube-api-access-cfqfr\") pod \"ovnkube-node-6vl2w\" (UID: \"72f650cd-84a0-41e8-b71e-982622024be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.359979 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.359945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" Jul 18 01:48:18.374022 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.373997 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" Jul 18 01:48:18.374022 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.374018 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 01:48:18.378113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.378085 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2r886" Jul 18 01:48:18.383012 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.382991 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" Jul 18 01:48:18.389900 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.389879 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ghg58" Jul 18 01:48:18.397932 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.397911 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49dkv" Jul 18 01:48:18.404594 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.404574 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-22pm6" Jul 18 01:48:18.411682 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.411620 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:18.417250 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.417223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:18.783700 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.783588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:18.783700 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.783671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:18.783981 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:18.783707 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:18.783981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.783777 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:18.783981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.783844 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:18.783981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.783864 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 01:48:18.783981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.783884 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 01:48:18.783981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.783895 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6r7fn for pod openshift-network-diagnostics/network-check-target-cxhwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:18.783981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.783873 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs podName:6d8e552e-32ad-4c02-8e89-bc9d00d093e5 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:19.783852216 +0000 UTC m=+4.115093201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs") pod "network-metrics-daemon-v4r5t" (UID: "6d8e552e-32ad-4c02-8e89-bc9d00d093e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:18.783981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.783924 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret podName:55d05391-7c5e-416f-a050-ed1b32053280 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:19.783906947 +0000 UTC m=+4.115147908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret") pod "global-pull-secret-syncer-ndt4g" (UID: "55d05391-7c5e-416f-a050-ed1b32053280") : object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:18.783981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:18.783940 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn podName:403160fb-538f-437d-ba5a-37ae0c7c89d9 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:19.783930046 +0000 UTC m=+4.115171013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r7fn" (UniqueName: "kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn") pod "network-check-target-cxhwx" (UID: "403160fb-538f-437d-ba5a-37ae0c7c89d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:19.093093 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:19.093061 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d052cd5_4ee5_4cdd_bd68_3ddeeb6f62d0.slice/crio-6e231b59c8751f303b509efb5df243c3ab13011db287f4d1eb5bd9a02b4626c4 WatchSource:0}: Error finding container 6e231b59c8751f303b509efb5df243c3ab13011db287f4d1eb5bd9a02b4626c4: Status 404 returned error can't find the container with id 6e231b59c8751f303b509efb5df243c3ab13011db287f4d1eb5bd9a02b4626c4 Jul 18 01:48:19.094432 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:19.094401 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3ad0e3_9c72_43c5_8d9f_b45eb96f42a8.slice/crio-4d9dc155c1ead973f44f943665e7403083c4d838fb5d30ddb3d62e549b85511a WatchSource:0}: Error finding container 4d9dc155c1ead973f44f943665e7403083c4d838fb5d30ddb3d62e549b85511a: Status 404 returned error can't find the container with id 4d9dc155c1ead973f44f943665e7403083c4d838fb5d30ddb3d62e549b85511a Jul 18 01:48:19.095126 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:19.095094 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6aef8f_9538_4ad9_8ce3_9463040b5054.slice/crio-ce7d0b8bd4dd493e1f307eb8186df56e3cc71f7f3a582d29188bbb94a3265627 WatchSource:0}: Error finding container ce7d0b8bd4dd493e1f307eb8186df56e3cc71f7f3a582d29188bbb94a3265627: Status 404 returned error can't find the container with id ce7d0b8bd4dd493e1f307eb8186df56e3cc71f7f3a582d29188bbb94a3265627 Jul 18 01:48:19.096293 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:19.096252 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf223dcd3_22b0_4b2e_aad8_cc9d66639fd1.slice/crio-d336e6130b631b347337ea717de6d6dd2cce8d4d009881c0760452c4d3dc459e WatchSource:0}: Error finding container d336e6130b631b347337ea717de6d6dd2cce8d4d009881c0760452c4d3dc459e: Status 404 returned error can't find the container with id d336e6130b631b347337ea717de6d6dd2cce8d4d009881c0760452c4d3dc459e Jul 18 01:48:19.097637 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:19.097536 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f650cd_84a0_41e8_b71e_982622024be0.slice/crio-8a2d21935bbabf5a3104017b8ea29e27d3cd4988b15b5cf9b8f5a69d22494ce3 WatchSource:0}: Error finding container 8a2d21935bbabf5a3104017b8ea29e27d3cd4988b15b5cf9b8f5a69d22494ce3: Status 404 returned error can't find the container with id 8a2d21935bbabf5a3104017b8ea29e27d3cd4988b15b5cf9b8f5a69d22494ce3 Jul 18 01:48:19.100607 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:19.100581 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3eb90f_655a_4e6e_acca_0467391032f1.slice/crio-8571bc6bdc09c742788eaca4b214728278d63c2b18a1a710d70bfdc8f4095eb4 WatchSource:0}: Error finding container 8571bc6bdc09c742788eaca4b214728278d63c2b18a1a710d70bfdc8f4095eb4: Status 404 returned error can't find the container with id 8571bc6bdc09c742788eaca4b214728278d63c2b18a1a710d70bfdc8f4095eb4 Jul 18 01:48:19.100926 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:19.100904 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75d2da2_02ba_40e4_8a9d_efe762a83e51.slice/crio-1cac8e4cc4b4405dcc9acbf07bf59f30b84962ce04b153c7be42817bffaade60 WatchSource:0}: Error finding container 1cac8e4cc4b4405dcc9acbf07bf59f30b84962ce04b153c7be42817bffaade60: Status 404 returned error can't find the container with id 1cac8e4cc4b4405dcc9acbf07bf59f30b84962ce04b153c7be42817bffaade60 Jul 18 01:48:19.105193 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.105143 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 01:43:18 +0000 UTC" deadline="2028-01-18 00:50:27.271179698 +0000 UTC" Jul 18 01:48:19.105256 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.105194 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13175h2m8.165989417s" Jul 18 01:48:19.201037 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.200775 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal" event={"ID":"38b08105c0224d562f53bf7954a3df13","Type":"ContainerStarted","Data":"2790db34441d30c1ec4c5cb1deba59ab6d5ad2da9c82e252ddb240b1a4f5041e"} Jul 18 01:48:19.203335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.203305 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-22pm6" event={"ID":"b4d10c59-9d5a-476b-8a8f-091d79787977","Type":"ContainerStarted","Data":"909af8aeb050b963531ab319282f3ccef6a0a1599da9a234877add9190f14f18"} Jul 18 01:48:19.204968 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.204944 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" event={"ID":"72f650cd-84a0-41e8-b71e-982622024be0","Type":"ContainerStarted","Data":"8a2d21935bbabf5a3104017b8ea29e27d3cd4988b15b5cf9b8f5a69d22494ce3"} Jul 18 01:48:19.206083 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.206059 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" event={"ID":"3d6aef8f-9538-4ad9-8ce3-9463040b5054","Type":"ContainerStarted","Data":"ce7d0b8bd4dd493e1f307eb8186df56e3cc71f7f3a582d29188bbb94a3265627"} Jul 18 01:48:19.207284 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.207264 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ghg58" event={"ID":"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8","Type":"ContainerStarted","Data":"4d9dc155c1ead973f44f943665e7403083c4d838fb5d30ddb3d62e549b85511a"} Jul 18 01:48:19.208436 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.208411 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" event={"ID":"440cebb3-367e-4f50-9418-46b7182e7520","Type":"ContainerStarted","Data":"6c6c34d407f0cf97af2391e831028d3120c7f81f4d52f8b5c91cb091deaac2ff"} Jul 18 01:48:19.211311 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.211288 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n6gnr" event={"ID":"f75d2da2-02ba-40e4-8a9d-efe762a83e51","Type":"ContainerStarted","Data":"1cac8e4cc4b4405dcc9acbf07bf59f30b84962ce04b153c7be42817bffaade60"} Jul 18 01:48:19.214331 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.214310 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" event={"ID":"4d3eb90f-655a-4e6e-acca-0467391032f1","Type":"ContainerStarted","Data":"8571bc6bdc09c742788eaca4b214728278d63c2b18a1a710d70bfdc8f4095eb4"} Jul 18 01:48:19.214734 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.214694 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-157.ec2.internal" podStartSLOduration=2.21467559 podStartE2EDuration="2.21467559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:19.2145364 +0000 UTC m=+3.545777385" watchObservedRunningTime="2026-07-18 01:48:19.21467559 +0000 UTC m=+3.545916575" Jul 18 01:48:19.215687 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.215666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2r886" event={"ID":"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1","Type":"ContainerStarted","Data":"d336e6130b631b347337ea717de6d6dd2cce8d4d009881c0760452c4d3dc459e"} Jul 18 01:48:19.217787 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.217768 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49dkv" event={"ID":"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0","Type":"ContainerStarted","Data":"6e231b59c8751f303b509efb5df243c3ab13011db287f4d1eb5bd9a02b4626c4"} Jul 18 01:48:19.792499 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.792406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:19.792499 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.792473 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:19.792743 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:19.792525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:19.792743 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:19.792691 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:19.792892 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:19.792761 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs podName:6d8e552e-32ad-4c02-8e89-bc9d00d093e5 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:21.792738972 +0000 UTC m=+6.123979960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs") pod "network-metrics-daemon-v4r5t" (UID: "6d8e552e-32ad-4c02-8e89-bc9d00d093e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:19.793245 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:19.793226 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 01:48:19.793328 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:19.793251 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 01:48:19.793328 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:19.793265 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6r7fn for pod openshift-network-diagnostics/network-check-target-cxhwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:19.793328 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:19.793312 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn podName:403160fb-538f-437d-ba5a-37ae0c7c89d9 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:21.793296549 +0000 UTC m=+6.124537513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r7fn" (UniqueName: "kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn") pod "network-check-target-cxhwx" (UID: "403160fb-538f-437d-ba5a-37ae0c7c89d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:19.793487 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:19.793376 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:19.793487 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:19.793411 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret podName:55d05391-7c5e-416f-a050-ed1b32053280 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:21.793399138 +0000 UTC m=+6.124640118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret") pod "global-pull-secret-syncer-ndt4g" (UID: "55d05391-7c5e-416f-a050-ed1b32053280") : object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:20.105923 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:20.105826 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 01:43:18 +0000 UTC" deadline="2027-12-21 19:42:28.31547912 +0000 UTC" Jul 18 01:48:20.105923 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:20.105874 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12521h54m8.209610775s" Jul 18 01:48:20.190339 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:20.190299 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:20.190513 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:20.190436 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:20.190513 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:20.190302 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:20.190631 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:20.190555 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:20.190950 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:20.190759 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:20.190950 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:20.190902 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:20.245849 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:20.244771 2582 generic.go:356] "Generic (PLEG): container finished" podID="cdf25076591eab9f29e707e0b8f3f259" containerID="9e4dc04c8770c0c3bae2d7773ac093620ffb8f7129a2147875dd876ebb24b788" exitCode=0 Jul 18 01:48:20.245849 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:20.245755 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" event={"ID":"cdf25076591eab9f29e707e0b8f3f259","Type":"ContainerDied","Data":"9e4dc04c8770c0c3bae2d7773ac093620ffb8f7129a2147875dd876ebb24b788"} Jul 18 01:48:21.278943 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:21.278896 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" event={"ID":"cdf25076591eab9f29e707e0b8f3f259","Type":"ContainerStarted","Data":"908b10b0ef05e443dc112a13c836f80febda816be94cc78c552ced1e5bff3019"} Jul 18 01:48:21.809071 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:21.809021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:21.809288 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:21.809109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:21.809288 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:21.809153 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:21.809412 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:21.809297 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:21.809412 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:21.809364 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret podName:55d05391-7c5e-416f-a050-ed1b32053280 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:25.809344459 +0000 UTC m=+10.140585429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret") pod "global-pull-secret-syncer-ndt4g" (UID: "55d05391-7c5e-416f-a050-ed1b32053280") : object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:21.809849 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:21.809794 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:21.809972 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:21.809870 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs podName:6d8e552e-32ad-4c02-8e89-bc9d00d093e5 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:25.809853808 +0000 UTC m=+10.141094775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs") pod "network-metrics-daemon-v4r5t" (UID: "6d8e552e-32ad-4c02-8e89-bc9d00d093e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:21.809972 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:21.809959 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 01:48:21.810116 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:21.809978 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 01:48:21.810116 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:21.809990 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6r7fn for pod openshift-network-diagnostics/network-check-target-cxhwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:21.810116 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:21.810027 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn podName:403160fb-538f-437d-ba5a-37ae0c7c89d9 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:25.810015526 +0000 UTC m=+10.141256488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r7fn" (UniqueName: "kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn") pod "network-check-target-cxhwx" (UID: "403160fb-538f-437d-ba5a-37ae0c7c89d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:22.191623 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:22.191532 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:22.191819 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:22.191683 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:22.192124 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:22.192099 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:22.192269 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:22.192244 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:22.192364 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:22.192321 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:22.192464 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:22.192444 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:24.190022 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:24.189935 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:24.190489 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:24.190098 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:24.190489 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:24.189935 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:24.190489 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:24.190464 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:24.190996 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:24.190730 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:24.190996 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:24.190892 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:25.842454 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:25.842412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:25.842822 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:25.842490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:25.842822 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:25.842519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:25.842822 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:25.842630 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:25.842822 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:25.842714 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret podName:55d05391-7c5e-416f-a050-ed1b32053280 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:33.842686381 +0000 UTC m=+18.173927357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret") pod "global-pull-secret-syncer-ndt4g" (UID: "55d05391-7c5e-416f-a050-ed1b32053280") : object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:25.842994 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:25.842975 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 01:48:25.843029 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:25.843002 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 01:48:25.843029 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:25.843015 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6r7fn for pod openshift-network-diagnostics/network-check-target-cxhwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:25.843088 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:25.843060 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn podName:403160fb-538f-437d-ba5a-37ae0c7c89d9 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:33.843045501 +0000 UTC m=+18.174286480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r7fn" (UniqueName: "kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn") pod "network-check-target-cxhwx" (UID: "403160fb-538f-437d-ba5a-37ae0c7c89d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:25.843126 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:25.843120 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:25.843187 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:25.843157 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs podName:6d8e552e-32ad-4c02-8e89-bc9d00d093e5 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:33.843144501 +0000 UTC m=+18.174385461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs") pod "network-metrics-daemon-v4r5t" (UID: "6d8e552e-32ad-4c02-8e89-bc9d00d093e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:26.189868 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:26.189833 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:26.190054 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:26.189933 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:26.190270 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:26.190253 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:26.190379 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:26.190360 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:26.190523 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:26.190498 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:26.190645 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:26.190585 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:28.189421 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:28.189386 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:28.189919 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:28.189531 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:28.189919 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:28.189596 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:28.192236 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:28.191842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:28.192236 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:28.191914 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:28.192700 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:28.189668 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:30.191753 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:30.191708 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:30.192165 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:30.191829 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:30.192225 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:30.192179 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:30.192266 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:30.192255 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:30.192311 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:30.192301 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:30.192387 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:30.192363 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:32.188652 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:32.188620 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:32.188652 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:32.188653 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:32.189113 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:32.188627 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:32.189113 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:32.188776 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:32.189113 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:32.188904 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:32.189113 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:32.188997 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:33.901640 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:33.901608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:33.901640 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:33.901652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:33.902211 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:33.901779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:33.902211 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:33.901798 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:33.902211 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:33.901821 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 01:48:33.902211 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:33.901845 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 01:48:33.902211 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:33.901855 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6r7fn for pod openshift-network-diagnostics/network-check-target-cxhwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:33.902211 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:33.901858 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret podName:55d05391-7c5e-416f-a050-ed1b32053280 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:49.901838761 +0000 UTC m=+34.233079741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret") pod "global-pull-secret-syncer-ndt4g" (UID: "55d05391-7c5e-416f-a050-ed1b32053280") : object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:33.902211 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:33.901873 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:33.902211 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:33.901895 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn podName:403160fb-538f-437d-ba5a-37ae0c7c89d9 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:49.901883514 +0000 UTC m=+34.233124491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r7fn" (UniqueName: "kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn") pod "network-check-target-cxhwx" (UID: "403160fb-538f-437d-ba5a-37ae0c7c89d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:33.902211 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:33.901913 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs podName:6d8e552e-32ad-4c02-8e89-bc9d00d093e5 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:49.901901213 +0000 UTC m=+34.233142204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs") pod "network-metrics-daemon-v4r5t" (UID: "6d8e552e-32ad-4c02-8e89-bc9d00d093e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:34.188608 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:34.188535 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:34.188781 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:34.188540 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:34.188781 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:34.188653 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:34.188781 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:34.188705 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:34.188781 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:34.188766 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:34.188991 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:34.188815 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:36.189480 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:36.189448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:36.189981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:36.189552 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:36.189981 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:36.189628 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:36.189981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:36.189721 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:36.189981 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:36.189775 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:36.189981 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:36.189910 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:38.189654 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:38.189229 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:38.189654 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:38.189347 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:38.189654 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:38.189231 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:38.189654 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:38.189386 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:38.189654 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:38.189472 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:38.189654 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:38.189555 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:39.319154 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.318662 2582 generic.go:356] "Generic (PLEG): container finished" podID="440cebb3-367e-4f50-9418-46b7182e7520" containerID="2e69b9c80081ee55b34a19e521f15d54e03e6475799cf240d7b616270aea7f41" exitCode=0 Jul 18 01:48:39.319154 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.318790 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" event={"ID":"440cebb3-367e-4f50-9418-46b7182e7520","Type":"ContainerDied","Data":"2e69b9c80081ee55b34a19e521f15d54e03e6475799cf240d7b616270aea7f41"} Jul 18 01:48:39.320885 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.320856 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n6gnr" event={"ID":"f75d2da2-02ba-40e4-8a9d-efe762a83e51","Type":"ContainerStarted","Data":"96a8c766da8c0a3b2f35358843115f6e876a509e39082a842a453da266e8e1d7"} Jul 18 01:48:39.322568 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.322543 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" event={"ID":"4d3eb90f-655a-4e6e-acca-0467391032f1","Type":"ContainerStarted","Data":"103f168d6381c7113e121495aea563dc76c06fb3bf89678530bc41054e6d19cd"} Jul 18 01:48:39.324225 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.324177 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49dkv" event={"ID":"2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0","Type":"ContainerStarted","Data":"fd349ce2e289bc12ebeb0c6dbc16d225305d2137c941001c47c9396b3157139c"} Jul 18 01:48:39.325833 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.325811 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-22pm6" event={"ID":"b4d10c59-9d5a-476b-8a8f-091d79787977","Type":"ContainerStarted","Data":"865a5438c254a535ec0f9af879250d47e7b3449f023454693b24e6088b839c93"} Jul 18 01:48:39.329296 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.329273 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" event={"ID":"72f650cd-84a0-41e8-b71e-982622024be0","Type":"ContainerStarted","Data":"4c7835d9d0d1158674285e3c71a7dc621806fee8eff0556a0d70f0422d927665"} Jul 18 01:48:39.329408 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.329345 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" event={"ID":"72f650cd-84a0-41e8-b71e-982622024be0","Type":"ContainerStarted","Data":"6d3a16929e54168e2e44add32fe9cacad470ff7ddbfafa14ab5998a83939f9a7"} Jul 18 01:48:39.329408 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.329397 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" event={"ID":"72f650cd-84a0-41e8-b71e-982622024be0","Type":"ContainerStarted","Data":"3db4754819623626b47daaa9ee284cfc83f19d06816570eb522cfc9ff2120d63"} Jul 18 01:48:39.329521 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.329431 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" event={"ID":"72f650cd-84a0-41e8-b71e-982622024be0","Type":"ContainerStarted","Data":"69ea56bbc54344ccbfd7c7754213eeca93d84befa662cf3ce439a35692aedbe2"} Jul 18 01:48:39.329521 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.329488 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" event={"ID":"72f650cd-84a0-41e8-b71e-982622024be0","Type":"ContainerStarted","Data":"739c29f07c70955bb4e17606d4c71e941ff2dfbf37e4c2af839e23d5ab574afa"} Jul 18 01:48:39.329597 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.329536 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" event={"ID":"72f650cd-84a0-41e8-b71e-982622024be0","Type":"ContainerStarted","Data":"cec5c2c09130a7c51737a4d52730cd507ffa89612930bf702766353fa2807f86"} Jul 18 01:48:39.330827 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.330795 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" event={"ID":"3d6aef8f-9538-4ad9-8ce3-9463040b5054","Type":"ContainerStarted","Data":"f79871fc89410122333978985552d4ea55b109049bb709421e39a96f9d1a86d6"} Jul 18 01:48:39.332499 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.332474 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ghg58" event={"ID":"3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8","Type":"ContainerStarted","Data":"38041064f9383da13e8dbe000238a4c8ccd4441c31199144818e772e3bfcefd9"} Jul 18 01:48:39.341494 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.341449 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-157.ec2.internal" podStartSLOduration=22.341432364 podStartE2EDuration="22.341432364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:21.296276082 +0000 UTC m=+5.627517070" watchObservedRunningTime="2026-07-18 01:48:39.341432364 +0000 UTC m=+23.672673348" Jul 18 01:48:39.358540 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.358497 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6ksp2" podStartSLOduration=3.898992018 podStartE2EDuration="23.358485569s" totalImagesPullingTime="19.459493551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:39.358439442 +0000 UTC m=+23.689680424" watchObservedRunningTime="2026-07-18 01:48:39.358485569 +0000 UTC m=+23.689726550" Jul 18 01:48:39.371834 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.371792 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ghg58" podStartSLOduration=3.914682594 podStartE2EDuration="23.371777048s" totalImagesPullingTime="19.457094454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:39.371727895 +0000 UTC m=+23.702968876" watchObservedRunningTime="2026-07-18 01:48:39.371777048 +0000 UTC m=+23.703018029" Jul 18 01:48:39.387804 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.387764 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-22pm6" podStartSLOduration=2.938591622 podStartE2EDuration="22.3877318s" totalImagesPullingTime="19.449140178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:39.387533943 +0000 UTC m=+23.718774926" watchObservedRunningTime="2026-07-18 01:48:39.3877318 +0000 UTC m=+23.718972781" Jul 18 01:48:39.401420 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.401377 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n6gnr" podStartSLOduration=11.029483343 podStartE2EDuration="22.40136361s" totalImagesPullingTime="11.371880267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:39.400828235 +0000 UTC m=+23.732069218" watchObservedRunningTime="2026-07-18 01:48:39.40136361 +0000 UTC m=+23.732604631" Jul 18 01:48:39.418668 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.418620 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-49dkv" podStartSLOduration=3.89055079 podStartE2EDuration="23.418604299s" totalImagesPullingTime="19.528053509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:39.418496474 +0000 UTC m=+23.749737459" watchObservedRunningTime="2026-07-18 01:48:39.418604299 +0000 UTC m=+23.749845281" Jul 18 01:48:39.962510 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:39.962485 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 18 01:48:40.147951 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.147784 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-18T01:48:39.962504104Z","UUID":"6a3680de-b596-4bc0-a402-cc40578e7dcb","Handler":null,"Name":"","Endpoint":""} Jul 18 01:48:40.149493 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.149471 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 18 01:48:40.149627 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.149513 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 18 01:48:40.188634 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.188602 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:40.188819 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.188639 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:40.188819 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:40.188714 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:40.188819 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.188603 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:40.188980 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:40.188873 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:40.188980 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:40.188949 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:40.337027 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.336946 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" event={"ID":"4d3eb90f-655a-4e6e-acca-0467391032f1","Type":"ContainerStarted","Data":"093e65016bbf39289fd378340ddeced612247bae1dc7b1cda1a7e608766d13c8"} Jul 18 01:48:40.339505 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.339472 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2r886" event={"ID":"f223dcd3-22b0-4b2e-aad8-cc9d66639fd1","Type":"ContainerStarted","Data":"baa8779596545725c6dab447e67a99746b200d8b68b0658cdbe2bd0536e0e205"} Jul 18 01:48:40.366502 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.366434 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2r886" podStartSLOduration=4.9132229899999995 podStartE2EDuration="24.366413924s" totalImagesPullingTime="19.453190934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:40.366241029 +0000 UTC m=+24.697482012" watchObservedRunningTime="2026-07-18 01:48:40.366413924 +0000 UTC m=+24.697654907" Jul 18 01:48:40.834191 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.834153 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:40.834880 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:40.834856 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:41.343637 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:41.343595 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" event={"ID":"4d3eb90f-655a-4e6e-acca-0467391032f1","Type":"ContainerStarted","Data":"87534b106d7e035e2cd199ef4bc012ae6446d937399cceb3aeec61f33e19aea1"} Jul 18 01:48:41.344409 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:41.344390 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:41.344566 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:41.344544 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n6gnr" Jul 18 01:48:41.360295 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:41.360197 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csr8z" podStartSLOduration=3.738305476 podStartE2EDuration="25.360158599s" totalImagesPullingTime="21.621853123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:41.359827284 +0000 UTC m=+25.691068263" watchObservedRunningTime="2026-07-18 01:48:41.360158599 +0000 UTC m=+25.691399575" Jul 18 01:48:42.189058 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:42.189028 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:42.189314 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:42.189028 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:42.189314 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:42.189135 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:42.189314 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:42.189036 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:42.189314 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:42.189229 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:42.189515 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:42.189313 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:42.349493 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:42.349449 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" event={"ID":"72f650cd-84a0-41e8-b71e-982622024be0","Type":"ContainerStarted","Data":"19121f2683c9b3e5e7a98303679a5b0884ab01b4742aa89d086ad3226b10ef57"} Jul 18 01:48:44.191861 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:44.191665 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:44.192589 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:44.191711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:44.192589 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:44.191957 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:44.192589 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:44.191721 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:44.192589 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:44.192004 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:44.192589 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:44.192111 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:44.357102 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:44.357066 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" event={"ID":"72f650cd-84a0-41e8-b71e-982622024be0","Type":"ContainerStarted","Data":"9f19059dfcd2fa42c99c884e1a2c82d0aefdf6ec65daf51bf9d43b2cd3c4c0bd"} Jul 18 01:48:44.357406 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:44.357378 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:44.358691 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:44.358668 2582 generic.go:356] "Generic (PLEG): container finished" podID="440cebb3-367e-4f50-9418-46b7182e7520" containerID="d1364d46c87b643d9f4323dab9f961cd351746461a08c77f22305fc7e863251b" exitCode=0 Jul 18 01:48:44.358801 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:44.358706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" event={"ID":"440cebb3-367e-4f50-9418-46b7182e7520","Type":"ContainerDied","Data":"d1364d46c87b643d9f4323dab9f961cd351746461a08c77f22305fc7e863251b"} Jul 18 01:48:44.374094 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:44.373955 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:44.400524 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:44.400475 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" podStartSLOduration=7.877793523 podStartE2EDuration="27.400462867s" totalImagesPullingTime="19.522669344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:44.400061129 +0000 UTC m=+28.731302112" watchObservedRunningTime="2026-07-18 01:48:44.400462867 +0000 UTC m=+28.731703882" Jul 18 01:48:45.361605 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:45.361578 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:45.362149 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:45.361646 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:45.379836 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:45.379816 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:48:45.648413 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:45.648308 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ndt4g"] Jul 18 01:48:45.648574 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:45.648444 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:45.648574 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:45.648555 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:45.651361 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:45.651335 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v4r5t"] Jul 18 01:48:45.651479 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:45.651451 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:45.651581 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:45.651558 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:45.652013 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:45.651983 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cxhwx"] Jul 18 01:48:45.652104 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:45.652066 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:45.652150 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:45.652131 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:46.365429 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:46.365393 2582 generic.go:356] "Generic (PLEG): container finished" podID="440cebb3-367e-4f50-9418-46b7182e7520" containerID="0ad8d79bbb62937657ebb5e421eba9981fefd0b3c0d3e6da490992d6b46b571e" exitCode=0 Jul 18 01:48:46.366081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:46.365479 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" event={"ID":"440cebb3-367e-4f50-9418-46b7182e7520","Type":"ContainerDied","Data":"0ad8d79bbb62937657ebb5e421eba9981fefd0b3c0d3e6da490992d6b46b571e"} Jul 18 01:48:47.189403 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:47.189367 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:47.189403 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:47.189396 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:47.189612 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:47.189382 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:47.189612 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:47.189485 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:47.189612 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:47.189551 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:47.189756 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:47.189659 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:48.372270 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:48.372100 2582 generic.go:356] "Generic (PLEG): container finished" podID="440cebb3-367e-4f50-9418-46b7182e7520" containerID="e2d857341696060ab7e5e46424f315c2d00e5e62c35bb8adcb96e4432977ba14" exitCode=0 Jul 18 01:48:48.372769 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:48.372171 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" event={"ID":"440cebb3-367e-4f50-9418-46b7182e7520","Type":"ContainerDied","Data":"e2d857341696060ab7e5e46424f315c2d00e5e62c35bb8adcb96e4432977ba14"} Jul 18 01:48:49.189103 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:49.189065 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:49.189263 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:49.189114 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:49.189263 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:49.189065 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:49.189263 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.189192 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:49.189402 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.189265 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:49.189402 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.189347 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:49.917970 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:49.917931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:49.917970 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:49.917971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:49.917970 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:49.918003 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:49.918559 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.918106 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:49.918559 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.918107 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:49.918559 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.918127 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 01:48:49.918559 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.918147 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 01:48:49.918559 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.918160 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6r7fn for pod openshift-network-diagnostics/network-check-target-cxhwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:49.918559 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.918174 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs podName:6d8e552e-32ad-4c02-8e89-bc9d00d093e5 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:21.918156524 +0000 UTC m=+66.249397484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs") pod "network-metrics-daemon-v4r5t" (UID: "6d8e552e-32ad-4c02-8e89-bc9d00d093e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 01:48:49.918559 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.918194 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret podName:55d05391-7c5e-416f-a050-ed1b32053280 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:21.918183846 +0000 UTC m=+66.249424807 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret") pod "global-pull-secret-syncer-ndt4g" (UID: "55d05391-7c5e-416f-a050-ed1b32053280") : object "kube-system"/"original-pull-secret" not registered Jul 18 01:48:49.918559 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:49.918227 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn podName:403160fb-538f-437d-ba5a-37ae0c7c89d9 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:21.918210676 +0000 UTC m=+66.249451650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r7fn" (UniqueName: "kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn") pod "network-check-target-cxhwx" (UID: "403160fb-538f-437d-ba5a-37ae0c7c89d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 01:48:51.188809 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:51.188768 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:51.188809 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:51.188803 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:51.189322 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:51.188803 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:51.189322 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:51.188883 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndt4g" podUID="55d05391-7c5e-416f-a050-ed1b32053280" Jul 18 01:48:51.189322 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:51.189001 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v4r5t" podUID="6d8e552e-32ad-4c02-8e89-bc9d00d093e5" Jul 18 01:48:51.189322 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:51.189086 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxhwx" podUID="403160fb-538f-437d-ba5a-37ae0c7c89d9" Jul 18 01:48:52.384713 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.384679 2582 generic.go:356] "Generic (PLEG): container finished" podID="440cebb3-367e-4f50-9418-46b7182e7520" containerID="0ae0119f51456ea82f51a8b75aa70ade364d73a3029b48f8b492ecd71c43a7a2" exitCode=0 Jul 18 01:48:52.385263 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.384760 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" event={"ID":"440cebb3-367e-4f50-9418-46b7182e7520","Type":"ContainerDied","Data":"0ae0119f51456ea82f51a8b75aa70ade364d73a3029b48f8b492ecd71c43a7a2"} Jul 18 01:48:52.481835 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.481810 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-157.ec2.internal" event="NodeReady" Jul 18 01:48:52.482022 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.482009 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 18 01:48:52.516065 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.516042 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b688f646c-bh4kc"] Jul 18 01:48:52.517684 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.517669 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.520534 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.520513 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 18 01:48:52.520765 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.520733 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 18 01:48:52.520946 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.520930 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tsvv8\"" Jul 18 01:48:52.521308 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.521296 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 18 01:48:52.526894 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.526879 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 18 01:48:52.528038 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.528017 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lcdmx"] Jul 18 01:48:52.529596 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.529581 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bfjfq"] Jul 18 01:48:52.529697 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.529683 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lcdmx" Jul 18 01:48:52.531054 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.531040 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.531888 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.531869 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 18 01:48:52.532074 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.532054 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 18 01:48:52.532153 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.532077 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4j2w9\"" Jul 18 01:48:52.532424 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.532407 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 18 01:48:52.533036 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.533008 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b688f646c-bh4kc"] Jul 18 01:48:52.533898 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.533869 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 18 01:48:52.534001 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.533924 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 18 01:48:52.534001 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.533995 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s6gqx\"" Jul 18 01:48:52.543225 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.543170 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lcdmx"] Jul 18 01:48:52.543948 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.543929 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bfjfq"] Jul 18 01:48:52.639491 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639407 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-image-registry-private-configuration\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.639491 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-bound-sa-token\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.639653 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639525 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.639653 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639564 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.639653 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-certificates\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.639653 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vtt\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-kube-api-access-c8vtt\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.639653 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639629 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c04b51a1-a506-4586-a8da-b11bc378a4da-cert\") pod \"ingress-canary-lcdmx\" (UID: \"c04b51a1-a506-4586-a8da-b11bc378a4da\") " pod="openshift-ingress-canary/ingress-canary-lcdmx" Jul 18 01:48:52.639653 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-tmp-dir\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.639850 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-installation-pull-secrets\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.639850 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-config-volume\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.639850 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-trusted-ca\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.639850 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45ts\" (UniqueName: \"kubernetes.io/projected/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-kube-api-access-l45ts\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.639850 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639769 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-ca-trust-extracted\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.639850 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.639787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlgrk\" (UniqueName: \"kubernetes.io/projected/c04b51a1-a506-4586-a8da-b11bc378a4da-kube-api-access-qlgrk\") pod \"ingress-canary-lcdmx\" (UID: \"c04b51a1-a506-4586-a8da-b11bc378a4da\") " pod="openshift-ingress-canary/ingress-canary-lcdmx" Jul 18 01:48:52.740781 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740722 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-image-registry-private-configuration\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.740781 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-bound-sa-token\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.740985 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.740985 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740840 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.740985 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-certificates\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.740985 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740876 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vtt\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-kube-api-access-c8vtt\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.740985 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740898 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c04b51a1-a506-4586-a8da-b11bc378a4da-cert\") pod \"ingress-canary-lcdmx\" (UID: \"c04b51a1-a506-4586-a8da-b11bc378a4da\") " pod="openshift-ingress-canary/ingress-canary-lcdmx" Jul 18 01:48:52.740985 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-tmp-dir\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.740985 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:52.740938 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 01:48:52.740985 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-installation-pull-secrets\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.740985 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.740976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-config-volume\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.741378 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:52.741011 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls podName:0d33b730-8ecf-43aa-a72e-b4d9f0ad6737 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:53.240992552 +0000 UTC m=+37.572233533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls") pod "dns-default-bfjfq" (UID: "0d33b730-8ecf-43aa-a72e-b4d9f0ad6737") : secret "dns-default-metrics-tls" not found Jul 18 01:48:52.741378 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.741035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-trusted-ca\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.741378 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.741065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45ts\" (UniqueName: \"kubernetes.io/projected/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-kube-api-access-l45ts\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.741378 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.741100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-ca-trust-extracted\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.741378 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.741125 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgrk\" (UniqueName: \"kubernetes.io/projected/c04b51a1-a506-4586-a8da-b11bc378a4da-kube-api-access-qlgrk\") pod \"ingress-canary-lcdmx\" (UID: \"c04b51a1-a506-4586-a8da-b11bc378a4da\") " pod="openshift-ingress-canary/ingress-canary-lcdmx" Jul 18 01:48:52.741378 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.741301 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-tmp-dir\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.741378 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:52.740941 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 01:48:52.741378 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:52.741348 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b688f646c-bh4kc: secret "image-registry-tls" not found Jul 18 01:48:52.741760 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:52.741431 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls podName:8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b nodeName:}" failed. No retries permitted until 2026-07-18 01:48:53.241413445 +0000 UTC m=+37.572654419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls") pod "image-registry-6b688f646c-bh4kc" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b") : secret "image-registry-tls" not found Jul 18 01:48:52.741760 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.741604 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-config-volume\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.741865 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.741774 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-ca-trust-extracted\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.742029 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.741995 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-certificates\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.742230 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.742209 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-trusted-ca\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.745139 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.745118 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c04b51a1-a506-4586-a8da-b11bc378a4da-cert\") pod \"ingress-canary-lcdmx\" (UID: \"c04b51a1-a506-4586-a8da-b11bc378a4da\") " pod="openshift-ingress-canary/ingress-canary-lcdmx" Jul 18 01:48:52.745239 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.745149 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-image-registry-private-configuration\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.745239 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.745149 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-installation-pull-secrets\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.763275 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.763246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgrk\" (UniqueName: \"kubernetes.io/projected/c04b51a1-a506-4586-a8da-b11bc378a4da-kube-api-access-qlgrk\") pod \"ingress-canary-lcdmx\" (UID: \"c04b51a1-a506-4586-a8da-b11bc378a4da\") " pod="openshift-ingress-canary/ingress-canary-lcdmx" Jul 18 01:48:52.763950 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.763933 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45ts\" (UniqueName: \"kubernetes.io/projected/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-kube-api-access-l45ts\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:52.764009 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.763957 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vtt\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-kube-api-access-c8vtt\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.764146 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.764129 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-bound-sa-token\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:52.867231 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.867191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lcdmx" Jul 18 01:48:52.995792 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:52.995610 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lcdmx"] Jul 18 01:48:52.998810 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:48:52.998778 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04b51a1_a506_4586_a8da_b11bc378a4da.slice/crio-56015e63b09e13ed6958632ea6e158bcd4dd40413f397d29c45c9bc2372a8377 WatchSource:0}: Error finding container 56015e63b09e13ed6958632ea6e158bcd4dd40413f397d29c45c9bc2372a8377: Status 404 returned error can't find the container with id 56015e63b09e13ed6958632ea6e158bcd4dd40413f397d29c45c9bc2372a8377 Jul 18 01:48:53.189356 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.189261 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:48:53.189356 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.189261 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:48:53.189583 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.189449 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:48:53.192052 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.192029 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 01:48:53.192384 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.192367 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2rqj2\"" Jul 18 01:48:53.192496 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.192427 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 01:48:53.192496 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.192437 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mcksx\"" Jul 18 01:48:53.192496 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.192443 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 01:48:53.192496 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.192427 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 01:48:53.243985 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.243951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:53.244079 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.243994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:53.244126 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:53.244088 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 01:48:53.244126 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:53.244095 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 01:48:53.244186 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:53.244159 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls podName:0d33b730-8ecf-43aa-a72e-b4d9f0ad6737 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:54.244144587 +0000 UTC m=+38.575385547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls") pod "dns-default-bfjfq" (UID: "0d33b730-8ecf-43aa-a72e-b4d9f0ad6737") : secret "dns-default-metrics-tls" not found Jul 18 01:48:53.244229 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:53.244098 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b688f646c-bh4kc: secret "image-registry-tls" not found Jul 18 01:48:53.244261 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:53.244236 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls podName:8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b nodeName:}" failed. No retries permitted until 2026-07-18 01:48:54.244224867 +0000 UTC m=+38.575465827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls") pod "image-registry-6b688f646c-bh4kc" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b") : secret "image-registry-tls" not found Jul 18 01:48:53.388186 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.388150 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lcdmx" event={"ID":"c04b51a1-a506-4586-a8da-b11bc378a4da","Type":"ContainerStarted","Data":"56015e63b09e13ed6958632ea6e158bcd4dd40413f397d29c45c9bc2372a8377"} Jul 18 01:48:53.391123 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.391094 2582 generic.go:356] "Generic (PLEG): container finished" podID="440cebb3-367e-4f50-9418-46b7182e7520" containerID="78fcf1328c3cae6ec12d9f1a596a914579fa4454414b5b169e032f7015c80c25" exitCode=0 Jul 18 01:48:53.391243 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:53.391141 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" event={"ID":"440cebb3-367e-4f50-9418-46b7182e7520","Type":"ContainerDied","Data":"78fcf1328c3cae6ec12d9f1a596a914579fa4454414b5b169e032f7015c80c25"} Jul 18 01:48:54.252029 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:54.251997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:54.252228 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:54.252042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:54.252228 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:54.252172 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 01:48:54.252228 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:54.252179 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 01:48:54.252389 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:54.252255 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls podName:0d33b730-8ecf-43aa-a72e-b4d9f0ad6737 nodeName:}" failed. No retries permitted until 2026-07-18 01:48:56.252234626 +0000 UTC m=+40.583475611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls") pod "dns-default-bfjfq" (UID: "0d33b730-8ecf-43aa-a72e-b4d9f0ad6737") : secret "dns-default-metrics-tls" not found Jul 18 01:48:54.252389 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:54.252185 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b688f646c-bh4kc: secret "image-registry-tls" not found Jul 18 01:48:54.252389 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:54.252337 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls podName:8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b nodeName:}" failed. No retries permitted until 2026-07-18 01:48:56.252321926 +0000 UTC m=+40.583562890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls") pod "image-registry-6b688f646c-bh4kc" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b") : secret "image-registry-tls" not found Jul 18 01:48:54.397907 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:54.397870 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" event={"ID":"440cebb3-367e-4f50-9418-46b7182e7520","Type":"ContainerStarted","Data":"daa10885f5f8eff80ea69b76494a3e86f1f5a4d591602876ed613242e7ac8226"} Jul 18 01:48:54.424292 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:54.424200 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lwdx9" podStartSLOduration=9.032433324 podStartE2EDuration="38.424181522s" totalImagesPullingTime="29.391748198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:54.4224191 +0000 UTC m=+38.753660082" watchObservedRunningTime="2026-07-18 01:48:54.424181522 +0000 UTC m=+38.755422504" Jul 18 01:48:56.268593 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:56.268418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:48:56.268593 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:56.268575 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 01:48:56.269123 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:56.268608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:48:56.269123 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:56.268646 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls podName:0d33b730-8ecf-43aa-a72e-b4d9f0ad6737 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:00.268625422 +0000 UTC m=+44.599866397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls") pod "dns-default-bfjfq" (UID: "0d33b730-8ecf-43aa-a72e-b4d9f0ad6737") : secret "dns-default-metrics-tls" not found Jul 18 01:48:56.269123 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:56.268683 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 01:48:56.269123 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:56.268696 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b688f646c-bh4kc: secret "image-registry-tls" not found Jul 18 01:48:56.269123 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:48:56.268763 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls podName:8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b nodeName:}" failed. No retries permitted until 2026-07-18 01:49:00.268730371 +0000 UTC m=+44.599971353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls") pod "image-registry-6b688f646c-bh4kc" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b") : secret "image-registry-tls" not found Jul 18 01:48:56.404517 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:56.404484 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lcdmx" event={"ID":"c04b51a1-a506-4586-a8da-b11bc378a4da","Type":"ContainerStarted","Data":"8de8b7b19d47aad9d797eac641c3501a4ac1496f3f1f01169bece763ae6927b9"} Jul 18 01:48:56.420559 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:48:56.420508 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lcdmx" podStartSLOduration=1.910602189 podStartE2EDuration="4.420491985s" totalImagesPullingTime="2.509889796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:48:56.420287317 +0000 UTC m=+40.751528299" watchObservedRunningTime="2026-07-18 01:48:56.420491985 +0000 UTC m=+40.751732967" Jul 18 01:49:00.263603 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.263567 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5tjcm"] Jul 18 01:49:00.293966 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.293939 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5tjcm"] Jul 18 01:49:00.294092 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.294054 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.296661 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.296638 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 18 01:49:00.297846 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.297830 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 18 01:49:00.297846 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.297845 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jz89w\"" Jul 18 01:49:00.297971 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.297877 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 18 01:49:00.297971 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.297923 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 18 01:49:00.299726 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.299704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:49:00.299829 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.299789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:49:00.299894 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:00.299846 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 01:49:00.299947 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:00.299917 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 01:49:00.299947 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:00.299926 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls podName:0d33b730-8ecf-43aa-a72e-b4d9f0ad6737 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:08.29990327 +0000 UTC m=+52.631144249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls") pod "dns-default-bfjfq" (UID: "0d33b730-8ecf-43aa-a72e-b4d9f0ad6737") : secret "dns-default-metrics-tls" not found Jul 18 01:49:00.299947 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:00.299930 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b688f646c-bh4kc: secret "image-registry-tls" not found Jul 18 01:49:00.300089 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:00.300009 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls podName:8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b nodeName:}" failed. No retries permitted until 2026-07-18 01:49:08.299992413 +0000 UTC m=+52.631233378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls") pod "image-registry-6b688f646c-bh4kc" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b") : secret "image-registry-tls" not found Jul 18 01:49:00.400411 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.400379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.400561 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.400460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-data-volume\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.400561 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.400513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-crio-socket\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.400682 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.400590 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.400682 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.400625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvjx\" (UniqueName: \"kubernetes.io/projected/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-api-access-9dvjx\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.501166 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.501135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.501325 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.501193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-data-volume\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.501325 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.501249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-crio-socket\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.501325 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.501268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.501325 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:00.501286 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 01:49:00.501492 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:00.501358 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls podName:aadd2bf8-b1fa-4a7f-bfa0-468ce296e214 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:01.001340894 +0000 UTC m=+45.332581854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5tjcm" (UID: "aadd2bf8-b1fa-4a7f-bfa0-468ce296e214") : secret "insights-runtime-extractor-tls" not found Jul 18 01:49:00.501492 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.501291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvjx\" (UniqueName: \"kubernetes.io/projected/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-api-access-9dvjx\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.501492 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:00.501374 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 01:49:00.501492 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:00.501450 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm podName:aadd2bf8-b1fa-4a7f-bfa0-468ce296e214 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:01.001433035 +0000 UTC m=+45.332674019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5tjcm" (UID: "aadd2bf8-b1fa-4a7f-bfa0-468ce296e214") : configmap "kube-rbac-proxy" not found Jul 18 01:49:00.501492 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.501467 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-crio-socket\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.501659 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.501617 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-data-volume\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:00.516094 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:00.516036 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvjx\" (UniqueName: \"kubernetes.io/projected/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-api-access-9dvjx\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:01.005026 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:01.004988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:01.005193 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:01.005093 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:01.005193 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:01.005125 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 01:49:01.005193 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:01.005166 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 01:49:01.005288 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:01.005197 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls podName:aadd2bf8-b1fa-4a7f-bfa0-468ce296e214 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:02.005180896 +0000 UTC m=+46.336421856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5tjcm" (UID: "aadd2bf8-b1fa-4a7f-bfa0-468ce296e214") : secret "insights-runtime-extractor-tls" not found Jul 18 01:49:01.005288 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:01.005211 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm podName:aadd2bf8-b1fa-4a7f-bfa0-468ce296e214 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:02.005204479 +0000 UTC m=+46.336445438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5tjcm" (UID: "aadd2bf8-b1fa-4a7f-bfa0-468ce296e214") : configmap "kube-rbac-proxy" not found Jul 18 01:49:02.012092 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:02.012053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:02.012596 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:02.012161 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:02.012596 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:02.012238 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 01:49:02.012596 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:02.012322 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls podName:aadd2bf8-b1fa-4a7f-bfa0-468ce296e214 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:04.012298509 +0000 UTC m=+48.343539468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5tjcm" (UID: "aadd2bf8-b1fa-4a7f-bfa0-468ce296e214") : secret "insights-runtime-extractor-tls" not found Jul 18 01:49:02.012596 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:02.012244 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 01:49:02.012596 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:02.012376 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm podName:aadd2bf8-b1fa-4a7f-bfa0-468ce296e214 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:04.012364345 +0000 UTC m=+48.343605306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5tjcm" (UID: "aadd2bf8-b1fa-4a7f-bfa0-468ce296e214") : configmap "kube-rbac-proxy" not found Jul 18 01:49:02.139562 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:02.139540 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-22pm6_b4d10c59-9d5a-476b-8a8f-091d79787977/dns-node-resolver/0.log" Jul 18 01:49:03.140153 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.140128 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ghg58_3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8/node-ca/0.log" Jul 18 01:49:03.366230 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.366203 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wshsq"] Jul 18 01:49:03.409941 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.409864 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wshsq"] Jul 18 01:49:03.410063 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.409959 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.413071 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.413052 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 18 01:49:03.413751 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.413726 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 18 01:49:03.413993 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.413979 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 18 01:49:03.414060 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.413992 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lqtpz\"" Jul 18 01:49:03.414111 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.414073 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 18 01:49:03.521834 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.521795 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8b9\" (UniqueName: \"kubernetes.io/projected/cafaf26f-487d-4cf8-9bf9-4d3231e2a66f-kube-api-access-pm8b9\") pod \"service-ca-58598695f7-wshsq\" (UID: \"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f\") " pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.521983 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.521921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cafaf26f-487d-4cf8-9bf9-4d3231e2a66f-signing-cabundle\") pod \"service-ca-58598695f7-wshsq\" (UID: \"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f\") " pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.522022 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.521987 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cafaf26f-487d-4cf8-9bf9-4d3231e2a66f-signing-key\") pod \"service-ca-58598695f7-wshsq\" (UID: \"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f\") " pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.622481 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.622441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cafaf26f-487d-4cf8-9bf9-4d3231e2a66f-signing-cabundle\") pod \"service-ca-58598695f7-wshsq\" (UID: \"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f\") " pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.622632 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.622530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cafaf26f-487d-4cf8-9bf9-4d3231e2a66f-signing-key\") pod \"service-ca-58598695f7-wshsq\" (UID: \"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f\") " pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.622632 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.622564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8b9\" (UniqueName: \"kubernetes.io/projected/cafaf26f-487d-4cf8-9bf9-4d3231e2a66f-kube-api-access-pm8b9\") pod \"service-ca-58598695f7-wshsq\" (UID: \"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f\") " pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.623095 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.623073 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cafaf26f-487d-4cf8-9bf9-4d3231e2a66f-signing-cabundle\") pod \"service-ca-58598695f7-wshsq\" (UID: \"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f\") " pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.624970 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.624951 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cafaf26f-487d-4cf8-9bf9-4d3231e2a66f-signing-key\") pod \"service-ca-58598695f7-wshsq\" (UID: \"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f\") " pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.634265 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.634235 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8b9\" (UniqueName: \"kubernetes.io/projected/cafaf26f-487d-4cf8-9bf9-4d3231e2a66f-kube-api-access-pm8b9\") pod \"service-ca-58598695f7-wshsq\" (UID: \"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f\") " pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.719046 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.718961 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wshsq" Jul 18 01:49:03.859849 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:03.859795 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wshsq"] Jul 18 01:49:03.863119 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:03.863092 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafaf26f_487d_4cf8_9bf9_4d3231e2a66f.slice/crio-3df059c4d419d564d51bb3302786487615b8f93425884cb9cf4d739ef0d6f9d4 WatchSource:0}: Error finding container 3df059c4d419d564d51bb3302786487615b8f93425884cb9cf4d739ef0d6f9d4: Status 404 returned error can't find the container with id 3df059c4d419d564d51bb3302786487615b8f93425884cb9cf4d739ef0d6f9d4 Jul 18 01:49:04.026643 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:04.026572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:04.026643 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:04.026615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:04.026852 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:04.026706 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 01:49:04.026852 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:04.026722 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 01:49:04.026852 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:04.026791 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls podName:aadd2bf8-b1fa-4a7f-bfa0-468ce296e214 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:08.026777397 +0000 UTC m=+52.358018358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5tjcm" (UID: "aadd2bf8-b1fa-4a7f-bfa0-468ce296e214") : secret "insights-runtime-extractor-tls" not found Jul 18 01:49:04.026852 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:04.026808 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm podName:aadd2bf8-b1fa-4a7f-bfa0-468ce296e214 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:08.026801376 +0000 UTC m=+52.358042335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5tjcm" (UID: "aadd2bf8-b1fa-4a7f-bfa0-468ce296e214") : configmap "kube-rbac-proxy" not found Jul 18 01:49:04.152992 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:04.152962 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lcdmx_c04b51a1-a506-4586-a8da-b11bc378a4da/serve-healthcheck-canary/0.log" Jul 18 01:49:04.422139 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:04.422105 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wshsq" event={"ID":"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f","Type":"ContainerStarted","Data":"3df059c4d419d564d51bb3302786487615b8f93425884cb9cf4d739ef0d6f9d4"} Jul 18 01:49:06.429630 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:06.429603 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wshsq" event={"ID":"cafaf26f-487d-4cf8-9bf9-4d3231e2a66f","Type":"ContainerStarted","Data":"14f65d8b6764016654a2b75e808a1e7af8d5f452560725c01c5bd4122bcfc946"} Jul 18 01:49:06.447151 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:06.447106 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-wshsq" podStartSLOduration=1.516914205 podStartE2EDuration="3.447088501s" totalImagesPullingTime="1.930174296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:06.446170257 +0000 UTC m=+50.777411252" watchObservedRunningTime="2026-07-18 01:49:06.447088501 +0000 UTC m=+50.778329484" Jul 18 01:49:08.060762 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:08.060708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:08.061235 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:08.060796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:08.061235 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:08.060926 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 01:49:08.061235 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:08.060986 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls podName:aadd2bf8-b1fa-4a7f-bfa0-468ce296e214 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:16.060968589 +0000 UTC m=+60.392209553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5tjcm" (UID: "aadd2bf8-b1fa-4a7f-bfa0-468ce296e214") : secret "insights-runtime-extractor-tls" not found Jul 18 01:49:08.061420 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:08.061351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:08.364191 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:08.364158 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:49:08.364439 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:08.364205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:49:08.364439 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:08.364309 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 01:49:08.364439 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:08.364319 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b688f646c-bh4kc: secret "image-registry-tls" not found Jul 18 01:49:08.364439 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:08.364321 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 01:49:08.364439 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:08.364373 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls podName:8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b nodeName:}" failed. No retries permitted until 2026-07-18 01:49:24.364359191 +0000 UTC m=+68.695600152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls") pod "image-registry-6b688f646c-bh4kc" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b") : secret "image-registry-tls" not found Jul 18 01:49:08.364439 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:08.364386 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls podName:0d33b730-8ecf-43aa-a72e-b4d9f0ad6737 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:24.364380127 +0000 UTC m=+68.695621087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls") pod "dns-default-bfjfq" (UID: "0d33b730-8ecf-43aa-a72e-b4d9f0ad6737") : secret "dns-default-metrics-tls" not found Jul 18 01:49:16.134552 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:16.134509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:16.136885 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:16.136857 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadd2bf8-b1fa-4a7f-bfa0-468ce296e214-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5tjcm\" (UID: \"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214\") " pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:16.205959 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:16.205933 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jz89w\"" Jul 18 01:49:16.213422 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:16.213399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5tjcm" Jul 18 01:49:16.341449 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:16.341412 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5tjcm"] Jul 18 01:49:16.345158 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:16.345128 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadd2bf8_b1fa_4a7f_bfa0_468ce296e214.slice/crio-1a6d27b03117bac17255de02a9188eb0aa2743dc1ef7b33ed23a2f19a3bf7e64 WatchSource:0}: Error finding container 1a6d27b03117bac17255de02a9188eb0aa2743dc1ef7b33ed23a2f19a3bf7e64: Status 404 returned error can't find the container with id 1a6d27b03117bac17255de02a9188eb0aa2743dc1ef7b33ed23a2f19a3bf7e64 Jul 18 01:49:16.455243 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:16.455162 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5tjcm" event={"ID":"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214","Type":"ContainerStarted","Data":"1a6d27b03117bac17255de02a9188eb0aa2743dc1ef7b33ed23a2f19a3bf7e64"} Jul 18 01:49:17.391356 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:17.391320 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vl2w" Jul 18 01:49:21.470525 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.470488 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5tjcm" event={"ID":"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214","Type":"ContainerStarted","Data":"68a1e60a9b44a624f2e06932b067d326c7d55d1ef2f97db6c6218f03999bd1e6"} Jul 18 01:49:21.985913 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.985880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:49:21.986105 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.985923 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:49:21.986105 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.986010 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:49:21.988610 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.988572 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 01:49:21.989447 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.989424 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 01:49:21.989562 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.989449 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 01:49:21.998716 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.998539 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 01:49:21.999531 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.999486 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8e552e-32ad-4c02-8e89-bc9d00d093e5-metrics-certs\") pod \"network-metrics-daemon-v4r5t\" (UID: \"6d8e552e-32ad-4c02-8e89-bc9d00d093e5\") " pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:49:21.999531 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:21.999490 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d05391-7c5e-416f-a050-ed1b32053280-original-pull-secret\") pod \"global-pull-secret-syncer-ndt4g\" (UID: \"55d05391-7c5e-416f-a050-ed1b32053280\") " pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:49:22.010723 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.010693 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r7fn\" (UniqueName: \"kubernetes.io/projected/403160fb-538f-437d-ba5a-37ae0c7c89d9-kube-api-access-6r7fn\") pod \"network-check-target-cxhwx\" (UID: \"403160fb-538f-437d-ba5a-37ae0c7c89d9\") " pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:49:22.011212 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.011191 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mcksx\"" Jul 18 01:49:22.019344 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.019322 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v4r5t" Jul 18 01:49:22.175479 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.175398 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v4r5t"] Jul 18 01:49:22.180852 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:22.180822 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8e552e_32ad_4c02_8e89_bc9d00d093e5.slice/crio-9bf1f67c1d4429857e9ae270a6c2d7447ceb7cef6a352b9d9a6cf54a991d171a WatchSource:0}: Error finding container 9bf1f67c1d4429857e9ae270a6c2d7447ceb7cef6a352b9d9a6cf54a991d171a: Status 404 returned error can't find the container with id 9bf1f67c1d4429857e9ae270a6c2d7447ceb7cef6a352b9d9a6cf54a991d171a Jul 18 01:49:22.299132 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.299009 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndt4g" Jul 18 01:49:22.307620 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.307586 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2rqj2\"" Jul 18 01:49:22.315653 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.315632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:49:22.451627 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.451592 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ndt4g"] Jul 18 01:49:22.455619 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:22.455585 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d05391_7c5e_416f_a050_ed1b32053280.slice/crio-3cdef4c82b498547c9fac43b67019f512b08782887bfc5ccbe06bb6ba91a8fa4 WatchSource:0}: Error finding container 3cdef4c82b498547c9fac43b67019f512b08782887bfc5ccbe06bb6ba91a8fa4: Status 404 returned error can't find the container with id 3cdef4c82b498547c9fac43b67019f512b08782887bfc5ccbe06bb6ba91a8fa4 Jul 18 01:49:22.466723 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.466698 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cxhwx"] Jul 18 01:49:22.469580 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:22.469551 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403160fb_538f_437d_ba5a_37ae0c7c89d9.slice/crio-c1b09ed80300e5fdbf9f76d34916cffbe5bdacdd6020d637e10d4c77d08dcec2 WatchSource:0}: Error finding container c1b09ed80300e5fdbf9f76d34916cffbe5bdacdd6020d637e10d4c77d08dcec2: Status 404 returned error can't find the container with id c1b09ed80300e5fdbf9f76d34916cffbe5bdacdd6020d637e10d4c77d08dcec2 Jul 18 01:49:22.474212 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.474127 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cxhwx" event={"ID":"403160fb-538f-437d-ba5a-37ae0c7c89d9","Type":"ContainerStarted","Data":"c1b09ed80300e5fdbf9f76d34916cffbe5bdacdd6020d637e10d4c77d08dcec2"} Jul 18 01:49:22.476623 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.476598 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v4r5t" event={"ID":"6d8e552e-32ad-4c02-8e89-bc9d00d093e5","Type":"ContainerStarted","Data":"9bf1f67c1d4429857e9ae270a6c2d7447ceb7cef6a352b9d9a6cf54a991d171a"} Jul 18 01:49:22.477807 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:22.477785 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ndt4g" event={"ID":"55d05391-7c5e-416f-a050-ed1b32053280","Type":"ContainerStarted","Data":"3cdef4c82b498547c9fac43b67019f512b08782887bfc5ccbe06bb6ba91a8fa4"} Jul 18 01:49:23.876063 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:23.876027 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-rkcvx"] Jul 18 01:49:23.946677 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:23.946640 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b688f646c-bh4kc"] Jul 18 01:49:23.946878 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:23.946707 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rkcvx"] Jul 18 01:49:23.946878 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:23.946860 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rkcvx" Jul 18 01:49:23.947012 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:23.946990 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" podUID="8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" Jul 18 01:49:23.952433 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:23.952398 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tc4rs\"" Jul 18 01:49:23.952665 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:23.952645 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 18 01:49:23.952942 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:23.952925 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 18 01:49:24.001968 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.001922 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjxq\" (UniqueName: \"kubernetes.io/projected/ce19db55-e204-4fa9-9e60-6eaca36ca929-kube-api-access-9cjxq\") pod \"downloads-5bc67dcd46-rkcvx\" (UID: \"ce19db55-e204-4fa9-9e60-6eaca36ca929\") " pod="openshift-console/downloads-5bc67dcd46-rkcvx" Jul 18 01:49:24.103261 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.103228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjxq\" (UniqueName: \"kubernetes.io/projected/ce19db55-e204-4fa9-9e60-6eaca36ca929-kube-api-access-9cjxq\") pod \"downloads-5bc67dcd46-rkcvx\" (UID: \"ce19db55-e204-4fa9-9e60-6eaca36ca929\") " pod="openshift-console/downloads-5bc67dcd46-rkcvx" Jul 18 01:49:24.119834 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.119800 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjxq\" (UniqueName: \"kubernetes.io/projected/ce19db55-e204-4fa9-9e60-6eaca36ca929-kube-api-access-9cjxq\") pod \"downloads-5bc67dcd46-rkcvx\" (UID: \"ce19db55-e204-4fa9-9e60-6eaca36ca929\") " pod="openshift-console/downloads-5bc67dcd46-rkcvx" Jul 18 01:49:24.260663 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.260581 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rkcvx" Jul 18 01:49:24.405982 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.405945 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:49:24.406155 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.406025 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:49:24.410073 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.410044 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d33b730-8ecf-43aa-a72e-b4d9f0ad6737-metrics-tls\") pod \"dns-default-bfjfq\" (UID: \"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737\") " pod="openshift-dns/dns-default-bfjfq" Jul 18 01:49:24.410210 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.410057 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"image-registry-6b688f646c-bh4kc\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:49:24.485281 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.485253 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:49:24.545405 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.545324 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:49:24.607666 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.607629 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") pod \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " Jul 18 01:49:24.607920 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.607679 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-bound-sa-token\") pod \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " Jul 18 01:49:24.607920 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.607718 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-ca-trust-extracted\") pod \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " Jul 18 01:49:24.608177 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.608156 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-certificates\") pod \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " Jul 18 01:49:24.608242 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.608201 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-installation-pull-secrets\") pod \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " Jul 18 01:49:24.608331 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.608303 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 01:49:24.608594 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.608571 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:49:24.608665 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.608648 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-image-registry-private-configuration\") pod \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " Jul 18 01:49:24.608804 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.608786 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8vtt\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-kube-api-access-c8vtt\") pod \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " Jul 18 01:49:24.608874 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.608820 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-trusted-ca\") pod \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\" (UID: \"8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b\") " Jul 18 01:49:24.609079 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.608970 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-ca-trust-extracted\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:49:24.609079 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.608988 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-certificates\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:49:24.609457 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.609434 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:49:24.611386 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.611328 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:49:24.611867 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.611829 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-kube-api-access-c8vtt" (OuterVolumeSpecName: "kube-api-access-c8vtt") pod "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b"). InnerVolumeSpecName "kube-api-access-c8vtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:49:24.612182 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.612106 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:49:24.613647 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.613623 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:49:24.617326 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.617303 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" (UID: "8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:49:24.674615 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.674588 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s6gqx\"" Jul 18 01:49:24.682726 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.682697 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bfjfq" Jul 18 01:49:24.709684 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.709654 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-registry-tls\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:49:24.709684 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.709682 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-bound-sa-token\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:49:24.709932 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.709695 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-installation-pull-secrets\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:49:24.709932 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.709710 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-image-registry-private-configuration\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:49:24.709932 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.709725 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vtt\" (UniqueName: \"kubernetes.io/projected/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-kube-api-access-c8vtt\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:49:24.709932 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:24.709759 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b-trusted-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:49:25.487647 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:25.487618 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b688f646c-bh4kc" Jul 18 01:49:25.523145 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:25.523107 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b688f646c-bh4kc"] Jul 18 01:49:25.528218 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:25.528194 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b688f646c-bh4kc"] Jul 18 01:49:26.192853 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:26.192822 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b" path="/var/lib/kubelet/pods/8d9fcecc-4d6a-4cdd-ae41-c0d21027b71b/volumes" Jul 18 01:49:28.616894 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:28.616811 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bfjfq"] Jul 18 01:49:28.621320 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:28.621289 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d33b730_8ecf_43aa_a72e_b4d9f0ad6737.slice/crio-21d33d8fb53491b288832a02b1a5b171db60135cc3ecf6d7895cae6a9e60b8cb WatchSource:0}: Error finding container 21d33d8fb53491b288832a02b1a5b171db60135cc3ecf6d7895cae6a9e60b8cb: Status 404 returned error can't find the container with id 21d33d8fb53491b288832a02b1a5b171db60135cc3ecf6d7895cae6a9e60b8cb Jul 18 01:49:28.638660 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:28.633103 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rkcvx"] Jul 18 01:49:28.642224 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:28.642187 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce19db55_e204_4fa9_9e60_6eaca36ca929.slice/crio-fc417c146c3804e9d27510da5303eafa53e092d950e67f6324b89a3efd6721d3 WatchSource:0}: Error finding container fc417c146c3804e9d27510da5303eafa53e092d950e67f6324b89a3efd6721d3: Status 404 returned error can't find the container with id fc417c146c3804e9d27510da5303eafa53e092d950e67f6324b89a3efd6721d3 Jul 18 01:49:29.507766 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.507713 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v4r5t" event={"ID":"6d8e552e-32ad-4c02-8e89-bc9d00d093e5","Type":"ContainerStarted","Data":"f1326da5edc135389cecd52b7fc888fb1bfb4214e0e756bb9a24b80e78d085e8"} Jul 18 01:49:29.507956 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.507796 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v4r5t" event={"ID":"6d8e552e-32ad-4c02-8e89-bc9d00d093e5","Type":"ContainerStarted","Data":"fb78954f4bba690bcadfda714cc2270e8d0b5dfe0ff9a0ed8806c5effad619c3"} Jul 18 01:49:29.510778 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.510687 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rkcvx" event={"ID":"ce19db55-e204-4fa9-9e60-6eaca36ca929","Type":"ContainerStarted","Data":"fc417c146c3804e9d27510da5303eafa53e092d950e67f6324b89a3efd6721d3"} Jul 18 01:49:29.513813 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.513707 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfjfq" event={"ID":"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737","Type":"ContainerStarted","Data":"21d33d8fb53491b288832a02b1a5b171db60135cc3ecf6d7895cae6a9e60b8cb"} Jul 18 01:49:29.515595 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.515566 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ndt4g" event={"ID":"55d05391-7c5e-416f-a050-ed1b32053280","Type":"ContainerStarted","Data":"e26067993dc7ad367c2adeae2294756e87cb80515a8b8c234113298d6cb3d63c"} Jul 18 01:49:29.517888 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.517829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5tjcm" event={"ID":"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214","Type":"ContainerStarted","Data":"d52dd505824eec23c308da562da1e14b808993b99d93c282ec34d152c87e686c"} Jul 18 01:49:29.520083 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.520029 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cxhwx" event={"ID":"403160fb-538f-437d-ba5a-37ae0c7c89d9","Type":"ContainerStarted","Data":"d4272e6bfe03180ed6cff024cc79dd0e5da68d2af8d2a84eae7c726d6d3c7606"} Jul 18 01:49:29.520215 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.520180 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:49:29.528649 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.528606 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v4r5t" podStartSLOduration=66.269142295 podStartE2EDuration="1m12.528588599s" totalImagesPullingTime="6.259446304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:29.527425405 +0000 UTC m=+73.858666388" watchObservedRunningTime="2026-07-18 01:49:29.528588599 +0000 UTC m=+73.859829581" Jul 18 01:49:29.546362 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.546314 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cxhwx" podStartSLOduration=66.572525828 podStartE2EDuration="1m12.546298842s" totalImagesPullingTime="5.973773014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:29.54594323 +0000 UTC m=+73.877184214" watchObservedRunningTime="2026-07-18 01:49:29.546298842 +0000 UTC m=+73.877539845" Jul 18 01:49:29.573793 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.573714 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ndt4g" podStartSLOduration=66.585550154 podStartE2EDuration="1m12.573692721s" totalImagesPullingTime="5.988142567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:29.572879401 +0000 UTC m=+73.904120385" watchObservedRunningTime="2026-07-18 01:49:29.573692721 +0000 UTC m=+73.904933684" Jul 18 01:49:29.802587 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.801822 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c56b984b7-7wd87"] Jul 18 01:49:29.805558 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.804224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.807360 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.806799 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tffsz\"" Jul 18 01:49:29.807360 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.807045 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 18 01:49:29.807360 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.807219 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 18 01:49:29.809075 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.807907 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 18 01:49:29.809075 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.808125 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 18 01:49:29.809075 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.808371 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 18 01:49:29.820935 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.820911 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c56b984b7-7wd87"] Jul 18 01:49:29.845414 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.845381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-oauth-serving-cert\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.845570 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.845435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-serving-cert\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.845570 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.845481 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-oauth-config\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.845570 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.845506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-console-config\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.845570 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.845550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8w8\" (UniqueName: \"kubernetes.io/projected/1a805c00-295b-4143-a18c-d425b029aa33-kube-api-access-cm8w8\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.845842 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.845577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-service-ca\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.946489 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.946387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-oauth-serving-cert\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.946489 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.946440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-serving-cert\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.947231 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.946528 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-oauth-config\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.947231 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.946554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-console-config\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.947231 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.946621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8w8\" (UniqueName: \"kubernetes.io/projected/1a805c00-295b-4143-a18c-d425b029aa33-kube-api-access-cm8w8\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.947231 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.946782 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-service-ca\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.947231 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.947143 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-oauth-serving-cert\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.947827 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.947413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-service-ca\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.947827 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.947623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-console-config\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.950835 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.950765 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-serving-cert\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.951294 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.951254 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-oauth-config\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:29.957382 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:29.957333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8w8\" (UniqueName: \"kubernetes.io/projected/1a805c00-295b-4143-a18c-d425b029aa33-kube-api-access-cm8w8\") pod \"console-c56b984b7-7wd87\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:30.117431 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:30.117141 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:30.511924 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:30.511650 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c56b984b7-7wd87"] Jul 18 01:49:30.526513 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:30.526482 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c56b984b7-7wd87" event={"ID":"1a805c00-295b-4143-a18c-d425b029aa33","Type":"ContainerStarted","Data":"42349c322f14d06e3d51616757ddeab8c3ad722ee533e85f1f3b1c0dd0f701cb"} Jul 18 01:49:30.529364 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:30.529339 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfjfq" event={"ID":"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737","Type":"ContainerStarted","Data":"46d3ec594d33293f3ba0ff12259b955ba639160f66ef762fc6a6b2c24e38049f"} Jul 18 01:49:31.536693 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:31.536651 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfjfq" event={"ID":"0d33b730-8ecf-43aa-a72e-b4d9f0ad6737","Type":"ContainerStarted","Data":"0616ffe99e506371d49227ad33b78f4fdf66cf964f478f66b65d1b65561868ab"} Jul 18 01:49:31.537267 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:31.537239 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bfjfq" Jul 18 01:49:31.557067 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:31.557011 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bfjfq" podStartSLOduration=37.795677082 podStartE2EDuration="39.556992375s" totalImagesPullingTime="1.761315293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:31.554059576 +0000 UTC m=+75.885300555" watchObservedRunningTime="2026-07-18 01:49:31.556992375 +0000 UTC m=+75.888233357" Jul 18 01:49:32.505245 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:32.504196 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c"] Jul 18 01:49:32.508499 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:32.508464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" Jul 18 01:49:32.511216 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:32.511086 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-brkxp\"" Jul 18 01:49:32.511325 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:32.511307 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 18 01:49:32.515272 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:32.515248 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c"] Jul 18 01:49:32.567409 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:32.567375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eda6f788-88a2-4aed-a1c1-bdabc978bfee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-46j8c\" (UID: \"eda6f788-88a2-4aed-a1c1-bdabc978bfee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" Jul 18 01:49:32.669102 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:32.668786 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eda6f788-88a2-4aed-a1c1-bdabc978bfee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-46j8c\" (UID: \"eda6f788-88a2-4aed-a1c1-bdabc978bfee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" Jul 18 01:49:32.669102 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:32.669070 2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 18 01:49:32.669301 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:32.669122 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eda6f788-88a2-4aed-a1c1-bdabc978bfee-tls-certificates podName:eda6f788-88a2-4aed-a1c1-bdabc978bfee nodeName:}" failed. No retries permitted until 2026-07-18 01:49:33.169105926 +0000 UTC m=+77.500346888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/eda6f788-88a2-4aed-a1c1-bdabc978bfee-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-46j8c" (UID: "eda6f788-88a2-4aed-a1c1-bdabc978bfee") : secret "prometheus-operator-admission-webhook-tls" not found Jul 18 01:49:33.173551 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:33.173506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eda6f788-88a2-4aed-a1c1-bdabc978bfee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-46j8c\" (UID: \"eda6f788-88a2-4aed-a1c1-bdabc978bfee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" Jul 18 01:49:33.176546 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:33.176517 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eda6f788-88a2-4aed-a1c1-bdabc978bfee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-46j8c\" (UID: \"eda6f788-88a2-4aed-a1c1-bdabc978bfee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" Jul 18 01:49:33.420755 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:33.420717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" Jul 18 01:49:34.532462 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:34.532394 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c"] Jul 18 01:49:34.536110 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:34.536080 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda6f788_88a2_4aed_a1c1_bdabc978bfee.slice/crio-6b9fc1d7427fabce2c2b49bb1034f7a9c9761cb9002d0b0acb45206b649fc927 WatchSource:0}: Error finding container 6b9fc1d7427fabce2c2b49bb1034f7a9c9761cb9002d0b0acb45206b649fc927: Status 404 returned error can't find the container with id 6b9fc1d7427fabce2c2b49bb1034f7a9c9761cb9002d0b0acb45206b649fc927 Jul 18 01:49:34.551733 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:34.551356 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c56b984b7-7wd87" event={"ID":"1a805c00-295b-4143-a18c-d425b029aa33","Type":"ContainerStarted","Data":"cd4df091dd8ee92cdb367310578bc2b5c2fa85b76f85f2d6f3869ddc002a04c2"} Jul 18 01:49:34.553322 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:34.553270 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" event={"ID":"eda6f788-88a2-4aed-a1c1-bdabc978bfee","Type":"ContainerStarted","Data":"6b9fc1d7427fabce2c2b49bb1034f7a9c9761cb9002d0b0acb45206b649fc927"} Jul 18 01:49:34.556100 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:34.556074 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5tjcm" event={"ID":"aadd2bf8-b1fa-4a7f-bfa0-468ce296e214","Type":"ContainerStarted","Data":"cc47c4633354e61827b613a900f96461bbeac39810817564575e0377f1af6953"} Jul 18 01:49:34.568806 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:34.568733 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c56b984b7-7wd87" podStartSLOduration=1.6938832069999998 podStartE2EDuration="5.568712787s" totalImagesPullingTime="3.87482958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:34.567877985 +0000 UTC m=+78.899118968" watchObservedRunningTime="2026-07-18 01:49:34.568712787 +0000 UTC m=+78.899953766" Jul 18 01:49:34.585866 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:34.585818 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5tjcm" podStartSLOduration=16.681545573 podStartE2EDuration="34.585803755s" totalImagesPullingTime="17.904258182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:49:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:34.584458703 +0000 UTC m=+78.915699685" watchObservedRunningTime="2026-07-18 01:49:34.585803755 +0000 UTC m=+78.917044738" Jul 18 01:49:36.565946 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:36.565826 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" event={"ID":"eda6f788-88a2-4aed-a1c1-bdabc978bfee","Type":"ContainerStarted","Data":"c68a5dee826478d8bb4e91a4cab8b5dbd6b6c5d75ea8dff15d59eb39369c7d97"} Jul 18 01:49:36.566393 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:36.566025 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" Jul 18 01:49:36.571318 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:36.571296 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" Jul 18 01:49:36.584789 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:36.584719 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-46j8c" podStartSLOduration=3.582545882 podStartE2EDuration="4.584703963s" totalImagesPullingTime="1.002158081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:36.583993862 +0000 UTC m=+80.915234849" watchObservedRunningTime="2026-07-18 01:49:36.584703963 +0000 UTC m=+80.915944944" Jul 18 01:49:37.575022 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.574987 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr"] Jul 18 01:49:37.580558 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.580533 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.583643 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.583622 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 18 01:49:37.584819 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.584783 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 18 01:49:37.584946 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.584884 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 18 01:49:37.585278 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.585120 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 18 01:49:37.585278 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.585124 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 18 01:49:37.593616 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.593599 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2bhq7\"" Jul 18 01:49:37.597834 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.597809 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr"] Jul 18 01:49:37.707609 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.707575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ceef4e-58cd-4918-a498-0db69ed5fe66-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.707813 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.707623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ceef4e-58cd-4918-a498-0db69ed5fe66-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.707813 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.707697 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pmf\" (UniqueName: \"kubernetes.io/projected/91ceef4e-58cd-4918-a498-0db69ed5fe66-kube-api-access-j5pmf\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.707947 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.707814 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ceef4e-58cd-4918-a498-0db69ed5fe66-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.809013 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.808975 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ceef4e-58cd-4918-a498-0db69ed5fe66-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.809201 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.809043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ceef4e-58cd-4918-a498-0db69ed5fe66-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.809201 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.809074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ceef4e-58cd-4918-a498-0db69ed5fe66-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.809201 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.809106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pmf\" (UniqueName: \"kubernetes.io/projected/91ceef4e-58cd-4918-a498-0db69ed5fe66-kube-api-access-j5pmf\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.809822 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.809794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ceef4e-58cd-4918-a498-0db69ed5fe66-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.812017 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.811978 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ceef4e-58cd-4918-a498-0db69ed5fe66-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.812017 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.811989 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ceef4e-58cd-4918-a498-0db69ed5fe66-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.817730 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.817707 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pmf\" (UniqueName: \"kubernetes.io/projected/91ceef4e-58cd-4918-a498-0db69ed5fe66-kube-api-access-j5pmf\") pod \"prometheus-operator-5945b5cfd-6nvkr\" (UID: \"91ceef4e-58cd-4918-a498-0db69ed5fe66\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:37.891495 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:37.891462 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" Jul 18 01:49:38.038949 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:38.038918 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr"] Jul 18 01:49:38.042414 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:38.042385 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ceef4e_58cd_4918_a498_0db69ed5fe66.slice/crio-9028c0ad2cd542e1ce460eca035c9f10249c6d8796964bf91008bed2eb122e7b WatchSource:0}: Error finding container 9028c0ad2cd542e1ce460eca035c9f10249c6d8796964bf91008bed2eb122e7b: Status 404 returned error can't find the container with id 9028c0ad2cd542e1ce460eca035c9f10249c6d8796964bf91008bed2eb122e7b Jul 18 01:49:38.575328 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:38.575291 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" event={"ID":"91ceef4e-58cd-4918-a498-0db69ed5fe66","Type":"ContainerStarted","Data":"9028c0ad2cd542e1ce460eca035c9f10249c6d8796964bf91008bed2eb122e7b"} Jul 18 01:49:40.117879 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:40.117841 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:40.118410 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:40.117956 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:40.123297 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:40.123201 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:40.586227 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:40.586150 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:49:41.545216 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:41.545186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bfjfq" Jul 18 01:49:42.258560 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:42.258523 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c56b984b7-7wd87"] Jul 18 01:49:47.614956 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:47.614914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rkcvx" event={"ID":"ce19db55-e204-4fa9-9e60-6eaca36ca929","Type":"ContainerStarted","Data":"bfe4649d15821b64caddd3ef439076121479ad94791e28a7f53bcf17780acf63"} Jul 18 01:49:47.615454 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:47.615374 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-rkcvx" Jul 18 01:49:47.617660 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:47.617632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" event={"ID":"91ceef4e-58cd-4918-a498-0db69ed5fe66","Type":"ContainerStarted","Data":"ec2c35015cbff8f2dd00b6975843f1282250289c5a8412ef5c0f68631368c981"} Jul 18 01:49:47.617794 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:47.617681 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" event={"ID":"91ceef4e-58cd-4918-a498-0db69ed5fe66","Type":"ContainerStarted","Data":"05cc5f16fd646f398679c47f940d86e87dcbb46be12f1b3c12e207e3c58642f4"} Jul 18 01:49:47.631466 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:47.631436 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-rkcvx" Jul 18 01:49:47.633026 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:47.632973 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-rkcvx" podStartSLOduration=6.531097868 podStartE2EDuration="24.632956102s" totalImagesPullingTime="18.101858234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:47.631886325 +0000 UTC m=+91.963127322" watchObservedRunningTime="2026-07-18 01:49:47.632956102 +0000 UTC m=+91.964197088" Jul 18 01:49:47.651445 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:47.651382 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6nvkr" podStartSLOduration=1.973247253 podStartE2EDuration="10.651362737s" totalImagesPullingTime="8.678115484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:47.65030478 +0000 UTC m=+91.981545764" watchObservedRunningTime="2026-07-18 01:49:47.651362737 +0000 UTC m=+91.982603723" Jul 18 01:49:49.893411 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.893376 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz"] Jul 18 01:49:49.916792 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.916727 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vvgnb"] Jul 18 01:49:49.916941 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.916850 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:49.919866 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.919844 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l9q6p\"" Jul 18 01:49:49.920320 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.920068 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 18 01:49:49.920631 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.920099 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 18 01:49:49.933692 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.933669 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-x9dzv"] Jul 18 01:49:49.933921 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.933792 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:49.936794 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.936762 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 18 01:49:49.937127 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.937008 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fc5s8\"" Jul 18 01:49:49.937267 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.937055 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 18 01:49:49.937365 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.937060 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 18 01:49:49.953982 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.953962 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz"] Jul 18 01:49:49.954235 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.954224 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-x9dzv"] Jul 18 01:49:49.954352 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.954168 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:49.958525 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.956931 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wms8k\"" Jul 18 01:49:49.958525 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.957200 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 18 01:49:49.958525 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.957526 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 18 01:49:49.958525 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:49.957858 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 18 01:49:50.007519 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.007486 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed19f185-adee-4f8a-a293-937d1dcd0df4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.007838 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.007808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed19f185-adee-4f8a-a293-937d1dcd0df4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.008092 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.008072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a7251de-e1da-4896-913d-6c002dbfe03f-root\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.008300 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.008281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wf4f\" (UniqueName: \"kubernetes.io/projected/0a7251de-e1da-4896-913d-6c002dbfe03f-kube-api-access-7wf4f\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.008490 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.008449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.008679 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.008642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a7251de-e1da-4896-913d-6c002dbfe03f-sys\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.008915 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.008896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmmf\" (UniqueName: \"kubernetes.io/projected/ed19f185-adee-4f8a-a293-937d1dcd0df4-kube-api-access-msmmf\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.009122 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.009105 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-wtmp\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.009296 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.009272 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.009465 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.009427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed19f185-adee-4f8a-a293-937d1dcd0df4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.009665 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.009613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a7251de-e1da-4896-913d-6c002dbfe03f-metrics-client-ca\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.009826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.009803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-textfile\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.010037 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.010020 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-tls\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.111007 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.110967 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-tls\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.111253 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.111236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed19f185-adee-4f8a-a293-937d1dcd0df4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.111360 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.111344 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rbh\" (UniqueName: \"kubernetes.io/projected/7502a389-1c73-417f-9f71-cf494086c9f1-kube-api-access-z2rbh\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.111473 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.111462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed19f185-adee-4f8a-a293-937d1dcd0df4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.112363 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.112343 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7502a389-1c73-417f-9f71-cf494086c9f1-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.112540 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.112527 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a7251de-e1da-4896-913d-6c002dbfe03f-root\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.112635 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.112622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.112730 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.112718 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wf4f\" (UniqueName: \"kubernetes.io/projected/0a7251de-e1da-4896-913d-6c002dbfe03f-kube-api-access-7wf4f\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.112856 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.112841 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.112952 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.112941 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a7251de-e1da-4896-913d-6c002dbfe03f-sys\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.113056 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113043 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7502a389-1c73-417f-9f71-cf494086c9f1-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.113148 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmmf\" (UniqueName: \"kubernetes.io/projected/ed19f185-adee-4f8a-a293-937d1dcd0df4-kube-api-access-msmmf\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.113224 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.112265 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed19f185-adee-4f8a-a293-937d1dcd0df4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.113406 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-wtmp\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.113550 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113528 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.113692 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.113838 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed19f185-adee-4f8a-a293-937d1dcd0df4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.113970 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a7251de-e1da-4896-913d-6c002dbfe03f-metrics-client-ca\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.114081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.114068 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-textfile\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.114209 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.114195 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.114391 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113835 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.115562 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.111268 2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 18 01:49:50.115772 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.114462 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a7251de-e1da-4896-913d-6c002dbfe03f-root\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.115900 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113340 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-wtmp\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.116005 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.113879 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a7251de-e1da-4896-913d-6c002dbfe03f-sys\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.116122 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.114959 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a7251de-e1da-4896-913d-6c002dbfe03f-metrics-client-ca\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.116234 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.115458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-textfile\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.116350 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.111360 2582 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 18 01:49:50.116803 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.116790 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-tls podName:0a7251de-e1da-4896-913d-6c002dbfe03f nodeName:}" failed. No retries permitted until 2026-07-18 01:49:50.615722874 +0000 UTC m=+94.946963846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-tls") pod "node-exporter-vvgnb" (UID: "0a7251de-e1da-4896-913d-6c002dbfe03f") : secret "node-exporter-tls" not found Jul 18 01:49:50.116972 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.116961 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed19f185-adee-4f8a-a293-937d1dcd0df4-openshift-state-metrics-tls podName:ed19f185-adee-4f8a-a293-937d1dcd0df4 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:50.61694139 +0000 UTC m=+94.948182357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ed19f185-adee-4f8a-a293-937d1dcd0df4-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-7r6xz" (UID: "ed19f185-adee-4f8a-a293-937d1dcd0df4") : secret "openshift-state-metrics-tls" not found Jul 18 01:49:50.117086 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.117025 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.121586 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.121547 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed19f185-adee-4f8a-a293-937d1dcd0df4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.121767 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.121704 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wf4f\" (UniqueName: \"kubernetes.io/projected/0a7251de-e1da-4896-913d-6c002dbfe03f-kube-api-access-7wf4f\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.124579 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.124555 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmmf\" (UniqueName: \"kubernetes.io/projected/ed19f185-adee-4f8a-a293-937d1dcd0df4-kube-api-access-msmmf\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.216765 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.215480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7502a389-1c73-417f-9f71-cf494086c9f1-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.216765 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.215542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.216765 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.215603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7502a389-1c73-417f-9f71-cf494086c9f1-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.216765 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.215637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.216765 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.215721 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.216765 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.215799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rbh\" (UniqueName: \"kubernetes.io/projected/7502a389-1c73-417f-9f71-cf494086c9f1-kube-api-access-z2rbh\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.216765 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.216539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7502a389-1c73-417f-9f71-cf494086c9f1-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.217605 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.217349 2582 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 18 01:49:50.217605 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.217417 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-tls podName:7502a389-1c73-417f-9f71-cf494086c9f1 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:50.717399528 +0000 UTC m=+95.048640510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-x9dzv" (UID: "7502a389-1c73-417f-9f71-cf494086c9f1") : secret "kube-state-metrics-tls" not found Jul 18 01:49:50.217605 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.217545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.217877 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.217613 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7502a389-1c73-417f-9f71-cf494086c9f1-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.222765 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.220211 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.233029 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.230092 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rbh\" (UniqueName: \"kubernetes.io/projected/7502a389-1c73-417f-9f71-cf494086c9f1-kube-api-access-z2rbh\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.619957 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.619921 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-tls\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:50.620243 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.620219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed19f185-adee-4f8a-a293-937d1dcd0df4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.620371 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.620071 2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 18 01:49:50.620514 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.620502 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-tls podName:0a7251de-e1da-4896-913d-6c002dbfe03f nodeName:}" failed. No retries permitted until 2026-07-18 01:49:51.620481376 +0000 UTC m=+95.951722352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-tls") pod "node-exporter-vvgnb" (UID: "0a7251de-e1da-4896-913d-6c002dbfe03f") : secret "node-exporter-tls" not found Jul 18 01:49:50.627071 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.626248 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed19f185-adee-4f8a-a293-937d1dcd0df4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-7r6xz\" (UID: \"ed19f185-adee-4f8a-a293-937d1dcd0df4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.721550 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.721506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:50.721767 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.721679 2582 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 18 01:49:50.721853 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:49:50.721774 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-tls podName:7502a389-1c73-417f-9f71-cf494086c9f1 nodeName:}" failed. No retries permitted until 2026-07-18 01:49:51.721753671 +0000 UTC m=+96.052994648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-x9dzv" (UID: "7502a389-1c73-417f-9f71-cf494086c9f1") : secret "kube-state-metrics-tls" not found Jul 18 01:49:50.833053 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.833017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" Jul 18 01:49:50.975293 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:50.972320 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 18 01:49:51.007372 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.007204 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 18 01:49:51.008149 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.007731 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.015255 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.014793 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 18 01:49:51.015255 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.015045 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 18 01:49:51.017688 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.017629 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 18 01:49:51.018766 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.018667 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 18 01:49:51.020537 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.020495 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 18 01:49:51.021084 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.021005 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 18 01:49:51.022345 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.021913 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 18 01:49:51.022345 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.022139 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-p2c9c\"" Jul 18 01:49:51.024409 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.023803 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 18 01:49:51.024409 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.024001 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 18 01:49:51.070245 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.070210 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz"] Jul 18 01:49:51.079342 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:51.079196 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded19f185_adee_4f8a_a293_937d1dcd0df4.slice/crio-542f32f6d10b013b6cf85f7f9fde222d0af9ff3785d7c7d61f0101be514153db WatchSource:0}: Error finding container 542f32f6d10b013b6cf85f7f9fde222d0af9ff3785d7c7d61f0101be514153db: Status 404 returned error can't find the container with id 542f32f6d10b013b6cf85f7f9fde222d0af9ff3785d7c7d61f0101be514153db Jul 18 01:49:51.125330 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125465 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125465 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-config-volume\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125607 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125607 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125607 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125556 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125607 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125610 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgs2\" (UniqueName: \"kubernetes.io/projected/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-kube-api-access-lwgs2\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-config-out\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125676 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125703 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-web-config\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.125852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.125788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226245 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226365 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226365 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226271 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgs2\" (UniqueName: \"kubernetes.io/projected/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-kube-api-access-lwgs2\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226365 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-config-out\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226365 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226325 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226365 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226341 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-web-config\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226634 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226634 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226634 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226634 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226634 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-config-volume\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226634 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226540 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.226634 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.226577 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.227664 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.227506 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.229792 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.229006 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.230407 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.230175 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.231785 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.231587 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.232753 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.232541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.233020 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.232949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.234270 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.233619 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.234270 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.233915 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-config-out\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.234270 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.234232 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.234478 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.234268 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.234478 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.234437 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-config-volume\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.234926 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.234882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-web-config\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.235675 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.235647 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgs2\" (UniqueName: \"kubernetes.io/projected/45a6eea2-53c2-4861-bfe3-884ffbbdbaef-kube-api-access-lwgs2\") pod \"alertmanager-main-0\" (UID: \"45a6eea2-53c2-4861-bfe3-884ffbbdbaef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.325485 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.325450 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 18 01:49:51.510532 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.510355 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 18 01:49:51.514718 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:51.514689 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a6eea2_53c2_4861_bfe3_884ffbbdbaef.slice/crio-05ed9df4d7149ce58bff561d6a3d601600296a5b5b3e97d0ccc7823efc3fd924 WatchSource:0}: Error finding container 05ed9df4d7149ce58bff561d6a3d601600296a5b5b3e97d0ccc7823efc3fd924: Status 404 returned error can't find the container with id 05ed9df4d7149ce58bff561d6a3d601600296a5b5b3e97d0ccc7823efc3fd924 Jul 18 01:49:51.631526 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.631487 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-tls\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:51.634415 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.634387 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a7251de-e1da-4896-913d-6c002dbfe03f-node-exporter-tls\") pod \"node-exporter-vvgnb\" (UID: \"0a7251de-e1da-4896-913d-6c002dbfe03f\") " pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:51.636990 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.636950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a6eea2-53c2-4861-bfe3-884ffbbdbaef","Type":"ContainerStarted","Data":"05ed9df4d7149ce58bff561d6a3d601600296a5b5b3e97d0ccc7823efc3fd924"} Jul 18 01:49:51.639232 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.639209 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" event={"ID":"ed19f185-adee-4f8a-a293-937d1dcd0df4","Type":"ContainerStarted","Data":"be523d03259c17015d0d96bd7db859ac250f44c9c553cbd2cfc777ea5c89a593"} Jul 18 01:49:51.639351 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.639253 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" event={"ID":"ed19f185-adee-4f8a-a293-937d1dcd0df4","Type":"ContainerStarted","Data":"909fe126ba47e2fb0c9a96a222a58d9163445300d58603061c7fb7064f54d4c5"} Jul 18 01:49:51.639351 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.639271 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" event={"ID":"ed19f185-adee-4f8a-a293-937d1dcd0df4","Type":"ContainerStarted","Data":"542f32f6d10b013b6cf85f7f9fde222d0af9ff3785d7c7d61f0101be514153db"} Jul 18 01:49:51.732976 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.732935 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:51.735957 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.735927 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7502a389-1c73-417f-9f71-cf494086c9f1-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-x9dzv\" (UID: \"7502a389-1c73-417f-9f71-cf494086c9f1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:51.746792 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.746759 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvgnb" Jul 18 01:49:51.767351 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.767334 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" Jul 18 01:49:51.777511 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:51.777475 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7251de_e1da_4896_913d_6c002dbfe03f.slice/crio-871e1f22551e805343ab56de14b998d1807ac9bdcb09b2b231d922b08bfaaffe WatchSource:0}: Error finding container 871e1f22551e805343ab56de14b998d1807ac9bdcb09b2b231d922b08bfaaffe: Status 404 returned error can't find the container with id 871e1f22551e805343ab56de14b998d1807ac9bdcb09b2b231d922b08bfaaffe Jul 18 01:49:51.968717 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:51.968547 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-x9dzv"] Jul 18 01:49:51.971540 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:51.971485 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7502a389_1c73_417f_9f71_cf494086c9f1.slice/crio-4dc793a03dc11c61b0147d713a62dedfd7adb52693c41a74f9041983e4c7345b WatchSource:0}: Error finding container 4dc793a03dc11c61b0147d713a62dedfd7adb52693c41a74f9041983e4c7345b: Status 404 returned error can't find the container with id 4dc793a03dc11c61b0147d713a62dedfd7adb52693c41a74f9041983e4c7345b Jul 18 01:49:52.645565 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:52.645522 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" event={"ID":"7502a389-1c73-417f-9f71-cf494086c9f1","Type":"ContainerStarted","Data":"4dc793a03dc11c61b0147d713a62dedfd7adb52693c41a74f9041983e4c7345b"} Jul 18 01:49:52.647468 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:52.647441 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvgnb" event={"ID":"0a7251de-e1da-4896-913d-6c002dbfe03f","Type":"ContainerStarted","Data":"871e1f22551e805343ab56de14b998d1807ac9bdcb09b2b231d922b08bfaaffe"} Jul 18 01:49:54.214403 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.214368 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bb6bf6858-gp2p4"] Jul 18 01:49:54.237988 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.236344 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bb6bf6858-gp2p4"] Jul 18 01:49:54.237988 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.236464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.240303 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.240010 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8x5qv\"" Jul 18 01:49:54.245677 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.241159 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 18 01:49:54.245677 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.241191 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cbc88ppqsrgh1\"" Jul 18 01:49:54.245677 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.241979 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 18 01:49:54.245677 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.242881 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 18 01:49:54.245677 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.242930 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 18 01:49:54.357686 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.357648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c681235c-786d-4e6b-bd53-923cac202780-secret-metrics-server-tls\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.358117 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.357695 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c681235c-786d-4e6b-bd53-923cac202780-audit-log\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.358117 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.357726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c681235c-786d-4e6b-bd53-923cac202780-secret-metrics-server-client-certs\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.358117 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.357809 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c681235c-786d-4e6b-bd53-923cac202780-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.358117 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.357907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c681235c-786d-4e6b-bd53-923cac202780-metrics-server-audit-profiles\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.358117 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.358025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c681235c-786d-4e6b-bd53-923cac202780-client-ca-bundle\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.358117 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.358074 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhfz\" (UniqueName: \"kubernetes.io/projected/c681235c-786d-4e6b-bd53-923cac202780-kube-api-access-bqhfz\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.459041 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.458662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c681235c-786d-4e6b-bd53-923cac202780-metrics-server-audit-profiles\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.459041 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.458728 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c681235c-786d-4e6b-bd53-923cac202780-client-ca-bundle\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.459041 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.458791 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhfz\" (UniqueName: \"kubernetes.io/projected/c681235c-786d-4e6b-bd53-923cac202780-kube-api-access-bqhfz\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.459041 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.458827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c681235c-786d-4e6b-bd53-923cac202780-secret-metrics-server-tls\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.459041 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.458853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c681235c-786d-4e6b-bd53-923cac202780-audit-log\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.459041 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.458879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c681235c-786d-4e6b-bd53-923cac202780-secret-metrics-server-client-certs\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.459041 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.458913 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c681235c-786d-4e6b-bd53-923cac202780-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.460207 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.460179 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c681235c-786d-4e6b-bd53-923cac202780-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.460442 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.460419 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c681235c-786d-4e6b-bd53-923cac202780-metrics-server-audit-profiles\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.460819 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.460791 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c681235c-786d-4e6b-bd53-923cac202780-audit-log\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.463907 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.463884 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c681235c-786d-4e6b-bd53-923cac202780-secret-metrics-server-client-certs\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.464127 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.463918 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c681235c-786d-4e6b-bd53-923cac202780-client-ca-bundle\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.464370 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.464327 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c681235c-786d-4e6b-bd53-923cac202780-secret-metrics-server-tls\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.479429 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.479396 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhfz\" (UniqueName: \"kubernetes.io/projected/c681235c-786d-4e6b-bd53-923cac202780-kube-api-access-bqhfz\") pod \"metrics-server-bb6bf6858-gp2p4\" (UID: \"c681235c-786d-4e6b-bd53-923cac202780\") " pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.553718 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.553677 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:49:54.701067 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.701034 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75985b5c64-mwmvz"] Jul 18 01:49:54.725755 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.722623 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75985b5c64-mwmvz"] Jul 18 01:49:54.725755 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.722778 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.734014 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.733989 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 18 01:49:54.773231 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.773206 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bb6bf6858-gp2p4"] Jul 18 01:49:54.864069 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.864001 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-console-config\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.864199 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.864080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-serving-cert\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.864199 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.864130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-oauth-serving-cert\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.864308 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.864201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-oauth-config\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.864308 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.864239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-trusted-ca-bundle\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.864308 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.864276 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-service-ca\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.864450 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.864316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54qtc\" (UniqueName: \"kubernetes.io/projected/7d17c69d-4292-4b7e-a14c-03ba72209903-kube-api-access-54qtc\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.893136 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:54.893106 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc681235c_786d_4e6b_bd53_923cac202780.slice/crio-18d19d07da6ddc0e1b8e564dab5d79c5bc8bac74bffdde2b6df00d16f731b46a WatchSource:0}: Error finding container 18d19d07da6ddc0e1b8e564dab5d79c5bc8bac74bffdde2b6df00d16f731b46a: Status 404 returned error can't find the container with id 18d19d07da6ddc0e1b8e564dab5d79c5bc8bac74bffdde2b6df00d16f731b46a Jul 18 01:49:54.965072 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.965047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-serving-cert\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.965180 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.965085 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-oauth-serving-cert\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.965180 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.965109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-oauth-config\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.965180 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.965131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-trusted-ca-bundle\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.965180 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.965168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-service-ca\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.965386 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.965211 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54qtc\" (UniqueName: \"kubernetes.io/projected/7d17c69d-4292-4b7e-a14c-03ba72209903-kube-api-access-54qtc\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.965386 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.965250 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-console-config\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.965961 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.965932 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-oauth-serving-cert\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.966087 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.966042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-service-ca\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.966799 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.966716 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-trusted-ca-bundle\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.967436 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.967396 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-console-config\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.968133 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.968111 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-oauth-config\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.968469 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.968450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-serving-cert\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:54.974048 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:54.974020 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54qtc\" (UniqueName: \"kubernetes.io/projected/7d17c69d-4292-4b7e-a14c-03ba72209903-kube-api-access-54qtc\") pod \"console-75985b5c64-mwmvz\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:55.066819 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.066786 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:49:55.272335 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.272274 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75985b5c64-mwmvz"] Jul 18 01:49:55.275890 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:49:55.275856 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d17c69d_4292_4b7e_a14c_03ba72209903.slice/crio-682e109353b1e228800b4a15569b6abd077c2f373f613049749509a2fbd91a82 WatchSource:0}: Error finding container 682e109353b1e228800b4a15569b6abd077c2f373f613049749509a2fbd91a82: Status 404 returned error can't find the container with id 682e109353b1e228800b4a15569b6abd077c2f373f613049749509a2fbd91a82 Jul 18 01:49:55.670212 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.670128 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75985b5c64-mwmvz" event={"ID":"7d17c69d-4292-4b7e-a14c-03ba72209903","Type":"ContainerStarted","Data":"6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1"} Jul 18 01:49:55.670212 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.670192 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75985b5c64-mwmvz" event={"ID":"7d17c69d-4292-4b7e-a14c-03ba72209903","Type":"ContainerStarted","Data":"682e109353b1e228800b4a15569b6abd077c2f373f613049749509a2fbd91a82"} Jul 18 01:49:55.671550 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.671520 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" event={"ID":"c681235c-786d-4e6b-bd53-923cac202780","Type":"ContainerStarted","Data":"18d19d07da6ddc0e1b8e564dab5d79c5bc8bac74bffdde2b6df00d16f731b46a"} Jul 18 01:49:55.673242 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.673210 2582 generic.go:356] "Generic (PLEG): container finished" podID="45a6eea2-53c2-4861-bfe3-884ffbbdbaef" containerID="7dff26fc547709572c87e6b982fc7c10abe56041a7d9e4100202dcd02e47fcfc" exitCode=0 Jul 18 01:49:55.673445 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.673415 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a6eea2-53c2-4861-bfe3-884ffbbdbaef","Type":"ContainerDied","Data":"7dff26fc547709572c87e6b982fc7c10abe56041a7d9e4100202dcd02e47fcfc"} Jul 18 01:49:55.676003 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.675982 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" event={"ID":"ed19f185-adee-4f8a-a293-937d1dcd0df4","Type":"ContainerStarted","Data":"f3ff3e35a1044c47a643d9ad123854a1d6677dee87fa5bdafdeab310eb682d3b"} Jul 18 01:49:55.678411 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.678386 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" event={"ID":"7502a389-1c73-417f-9f71-cf494086c9f1","Type":"ContainerStarted","Data":"eff65790e73a0ed0a1c50d98fb02ca364cf28d7f72c56c48d00d7b66c8f78051"} Jul 18 01:49:55.678517 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.678444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" event={"ID":"7502a389-1c73-417f-9f71-cf494086c9f1","Type":"ContainerStarted","Data":"5af051484b42de18ba41c4632da46c9b7f7d468d150a97e0c981804116852b0d"} Jul 18 01:49:55.678517 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.678476 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" event={"ID":"7502a389-1c73-417f-9f71-cf494086c9f1","Type":"ContainerStarted","Data":"ddfcf556e71feb6059edf66e4c427c1e9319e29e9a8dcdda46ff8355adf55b46"} Jul 18 01:49:55.680254 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.680228 2582 generic.go:356] "Generic (PLEG): container finished" podID="0a7251de-e1da-4896-913d-6c002dbfe03f" containerID="7de71b5dd988f41d10464f96606d1806ced8791adf3eaca7035142d413b4f33f" exitCode=0 Jul 18 01:49:55.680352 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.680265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvgnb" event={"ID":"0a7251de-e1da-4896-913d-6c002dbfe03f","Type":"ContainerDied","Data":"7de71b5dd988f41d10464f96606d1806ced8791adf3eaca7035142d413b4f33f"} Jul 18 01:49:55.687429 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.687384 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75985b5c64-mwmvz" podStartSLOduration=1.687368471 podStartE2EDuration="1.687368471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:55.686605724 +0000 UTC m=+100.017846706" watchObservedRunningTime="2026-07-18 01:49:55.687368471 +0000 UTC m=+100.018609448" Jul 18 01:49:55.706919 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.706702 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-x9dzv" podStartSLOduration=3.808674634 podStartE2EDuration="6.706682615s" totalImagesPullingTime="2.898007981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:55.704713541 +0000 UTC m=+100.035954528" watchObservedRunningTime="2026-07-18 01:49:55.706682615 +0000 UTC m=+100.037923596" Jul 18 01:49:55.722534 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:55.722410 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7r6xz" podStartSLOduration=4.009260139 podStartE2EDuration="6.72239498s" totalImagesPullingTime="2.713134841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:55.72137871 +0000 UTC m=+100.052619692" watchObservedRunningTime="2026-07-18 01:49:55.72239498 +0000 UTC m=+100.053635963" Jul 18 01:49:56.686454 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:56.686404 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvgnb" event={"ID":"0a7251de-e1da-4896-913d-6c002dbfe03f","Type":"ContainerStarted","Data":"6e9318f4e06f4f4550990d327cfd4a003f78327fae4ce8c5d04ff297ab409044"} Jul 18 01:49:56.686962 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:56.686470 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvgnb" event={"ID":"0a7251de-e1da-4896-913d-6c002dbfe03f","Type":"ContainerStarted","Data":"c1548dcc18ec766b83098f569078c915509749aa02fe2aac40787d9e32021231"} Jul 18 01:49:56.707904 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:56.707854 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vvgnb" podStartSLOduration=5.295977934 podStartE2EDuration="7.707840095s" totalImagesPullingTime="2.293753188s" totalInitContainerRuntime="118.108973ms" isStatefulPod=true podCreationTimestamp="2026-07-18 01:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:56.706677262 +0000 UTC m=+101.037918241" watchObservedRunningTime="2026-07-18 01:49:56.707840095 +0000 UTC m=+101.039081076" Jul 18 01:49:57.692950 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:57.692904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" event={"ID":"c681235c-786d-4e6b-bd53-923cac202780","Type":"ContainerStarted","Data":"9bef896950de8c46d4fda2f89990f6e296cda2e0bc3cbde8875706c12d08b98e"} Jul 18 01:49:57.711208 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:57.711157 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" podStartSLOduration=1.689677364 podStartE2EDuration="3.711140549s" totalImagesPullingTime="2.021463185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:49:57.709148283 +0000 UTC m=+102.040389265" watchObservedRunningTime="2026-07-18 01:49:57.711140549 +0000 UTC m=+102.042381530" Jul 18 01:49:58.698613 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:58.698583 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a6eea2-53c2-4861-bfe3-884ffbbdbaef","Type":"ContainerStarted","Data":"0663e5172d1882c2192fbc2ee32ac96556d7394140a1af4e1479c6d799173f2c"} Jul 18 01:49:58.698960 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:58.698661 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a6eea2-53c2-4861-bfe3-884ffbbdbaef","Type":"ContainerStarted","Data":"5d0369051aee587f1a178d5ccc92bc3f7b52e086b4a0394683cefc8da2fee29a"} Jul 18 01:49:59.707240 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:59.707202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a6eea2-53c2-4861-bfe3-884ffbbdbaef","Type":"ContainerStarted","Data":"8e76bd8fff739da4e6c76fda74d25d7163fa6d5d02cbb95d6bea038d7803df6e"} Jul 18 01:49:59.707673 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:59.707273 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a6eea2-53c2-4861-bfe3-884ffbbdbaef","Type":"ContainerStarted","Data":"5dc9acb1148f4bdc40bac62e14db95703b05e1e3c6e0222e8b7f505a3af343a3"} Jul 18 01:49:59.707673 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:49:59.707299 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a6eea2-53c2-4861-bfe3-884ffbbdbaef","Type":"ContainerStarted","Data":"0f70780cec62f1f6c9880f18b081d4a51fcf48f1648429c8d0a38c7e7ed43100"} Jul 18 01:50:00.532909 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:00.532836 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cxhwx" Jul 18 01:50:00.714446 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:00.714414 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a6eea2-53c2-4861-bfe3-884ffbbdbaef","Type":"ContainerStarted","Data":"810fd7a705c49998b9c8f0c588dff409602b88f8364aec714b5d4edabb614f1b"} Jul 18 01:50:00.739886 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:00.739841 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.321011898 podStartE2EDuration="10.739825484s" totalImagesPullingTime="6.393209989s" totalInitContainerRuntime="25.603597ms" isStatefulPod=false podCreationTimestamp="2026-07-18 01:49:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:50:00.738931641 +0000 UTC m=+105.070172625" watchObservedRunningTime="2026-07-18 01:50:00.739825484 +0000 UTC m=+105.071066465" Jul 18 01:50:05.067462 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:05.067422 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:50:05.067981 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:05.067487 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:50:05.072222 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:05.072201 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:50:05.738699 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:05.738669 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:50:07.611308 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.611240 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c56b984b7-7wd87" podUID="1a805c00-295b-4143-a18c-d425b029aa33" containerName="console" containerID="cri-o://cd4df091dd8ee92cdb367310578bc2b5c2fa85b76f85f2d6f3869ddc002a04c2" gracePeriod=15 Jul 18 01:50:07.742917 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.742888 2582 generic.go:356] "Generic (PLEG): container finished" podID="1a805c00-295b-4143-a18c-d425b029aa33" containerID="cd4df091dd8ee92cdb367310578bc2b5c2fa85b76f85f2d6f3869ddc002a04c2" exitCode=0 Jul 18 01:50:07.743027 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.742938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c56b984b7-7wd87" event={"ID":"1a805c00-295b-4143-a18c-d425b029aa33","Type":"ContainerDied","Data":"cd4df091dd8ee92cdb367310578bc2b5c2fa85b76f85f2d6f3869ddc002a04c2"} Jul 18 01:50:07.882316 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.882245 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:50:07.975502 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.975470 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-service-ca\") pod \"1a805c00-295b-4143-a18c-d425b029aa33\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " Jul 18 01:50:07.975676 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.975520 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm8w8\" (UniqueName: \"kubernetes.io/projected/1a805c00-295b-4143-a18c-d425b029aa33-kube-api-access-cm8w8\") pod \"1a805c00-295b-4143-a18c-d425b029aa33\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " Jul 18 01:50:07.975676 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.975543 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-serving-cert\") pod \"1a805c00-295b-4143-a18c-d425b029aa33\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " Jul 18 01:50:07.975676 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.975574 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-console-config\") pod \"1a805c00-295b-4143-a18c-d425b029aa33\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " Jul 18 01:50:07.975676 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.975595 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-oauth-config\") pod \"1a805c00-295b-4143-a18c-d425b029aa33\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " Jul 18 01:50:07.975676 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.975636 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-oauth-serving-cert\") pod \"1a805c00-295b-4143-a18c-d425b029aa33\" (UID: \"1a805c00-295b-4143-a18c-d425b029aa33\") " Jul 18 01:50:07.975966 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.975945 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-service-ca" (OuterVolumeSpecName: "service-ca") pod "1a805c00-295b-4143-a18c-d425b029aa33" (UID: "1a805c00-295b-4143-a18c-d425b029aa33"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:50:07.976071 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.976035 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-console-config" (OuterVolumeSpecName: "console-config") pod "1a805c00-295b-4143-a18c-d425b029aa33" (UID: "1a805c00-295b-4143-a18c-d425b029aa33"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:50:07.976139 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.976124 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1a805c00-295b-4143-a18c-d425b029aa33" (UID: "1a805c00-295b-4143-a18c-d425b029aa33"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:50:07.977992 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.977964 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1a805c00-295b-4143-a18c-d425b029aa33" (UID: "1a805c00-295b-4143-a18c-d425b029aa33"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:50:07.977992 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.977983 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a805c00-295b-4143-a18c-d425b029aa33-kube-api-access-cm8w8" (OuterVolumeSpecName: "kube-api-access-cm8w8") pod "1a805c00-295b-4143-a18c-d425b029aa33" (UID: "1a805c00-295b-4143-a18c-d425b029aa33"). InnerVolumeSpecName "kube-api-access-cm8w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:50:07.978266 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:07.978049 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1a805c00-295b-4143-a18c-d425b029aa33" (UID: "1a805c00-295b-4143-a18c-d425b029aa33"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:50:08.076761 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.076719 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-oauth-serving-cert\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:50:08.076761 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.076767 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:50:08.076966 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.076781 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8w8\" (UniqueName: \"kubernetes.io/projected/1a805c00-295b-4143-a18c-d425b029aa33-kube-api-access-cm8w8\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:50:08.076966 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.076793 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-serving-cert\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:50:08.076966 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.076805 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a805c00-295b-4143-a18c-d425b029aa33-console-config\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:50:08.076966 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.076816 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a805c00-295b-4143-a18c-d425b029aa33-console-oauth-config\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:50:08.748347 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.748315 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c56b984b7-7wd87" Jul 18 01:50:08.748347 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.748342 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c56b984b7-7wd87" event={"ID":"1a805c00-295b-4143-a18c-d425b029aa33","Type":"ContainerDied","Data":"42349c322f14d06e3d51616757ddeab8c3ad722ee533e85f1f3b1c0dd0f701cb"} Jul 18 01:50:08.748929 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.748408 2582 scope.go:117] "RemoveContainer" containerID="cd4df091dd8ee92cdb367310578bc2b5c2fa85b76f85f2d6f3869ddc002a04c2" Jul 18 01:50:08.770477 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.770452 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c56b984b7-7wd87"] Jul 18 01:50:08.776691 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:08.776669 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c56b984b7-7wd87"] Jul 18 01:50:10.193395 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:10.193362 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a805c00-295b-4143-a18c-d425b029aa33" path="/var/lib/kubelet/pods/1a805c00-295b-4143-a18c-d425b029aa33/volumes" Jul 18 01:50:14.554073 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:14.554045 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:50:14.554511 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:14.554098 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:50:16.438254 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:16.438225 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lcdmx_c04b51a1-a506-4586-a8da-b11bc378a4da/serve-healthcheck-canary/0.log" Jul 18 01:50:34.559531 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:34.559494 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:50:34.563392 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:50:34.563359 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bb6bf6858-gp2p4" Jul 18 01:51:14.204903 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.204817 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-66b85cb67-z74gv"] Jul 18 01:51:14.205587 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.205467 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a805c00-295b-4143-a18c-d425b029aa33" containerName="console" Jul 18 01:51:14.205587 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.205488 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a805c00-295b-4143-a18c-d425b029aa33" containerName="console" Jul 18 01:51:14.205807 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.205607 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a805c00-295b-4143-a18c-d425b029aa33" containerName="console" Jul 18 01:51:14.208688 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.208660 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.211551 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.211526 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8bn8f\"" Jul 18 01:51:14.211675 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.211565 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 18 01:51:14.211675 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.211580 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 18 01:51:14.211816 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.211674 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 18 01:51:14.211816 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.211809 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 18 01:51:14.211919 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.211873 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 18 01:51:14.218777 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.218122 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 18 01:51:14.219597 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.219577 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66b85cb67-z74gv"] Jul 18 01:51:14.309127 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.309093 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.309127 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.309130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-metrics-client-ca\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.309384 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.309153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-telemeter-client-tls\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.309384 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.309258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-serving-certs-ca-bundle\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.309384 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.309298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxcf4\" (UniqueName: \"kubernetes.io/projected/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-kube-api-access-kxcf4\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.309384 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.309328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.309384 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.309377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-federate-client-tls\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.309592 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.309409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-secret-telemeter-client\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.410130 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.410092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-telemeter-client-tls\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.410130 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.410132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-serving-certs-ca-bundle\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.410374 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.410153 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxcf4\" (UniqueName: \"kubernetes.io/projected/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-kube-api-access-kxcf4\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.410374 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.410170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.410374 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.410200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-federate-client-tls\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.410528 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.410384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-secret-telemeter-client\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.410528 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.410459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.410528 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.410512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-metrics-client-ca\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.411116 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.411092 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.411238 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.411119 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-serving-certs-ca-bundle\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.411238 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.411164 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-metrics-client-ca\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.412808 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.412787 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-federate-client-tls\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.412908 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.412827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-telemeter-client-tls\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.413369 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.413341 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.413532 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.413511 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-secret-telemeter-client\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.418159 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.418132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxcf4\" (UniqueName: \"kubernetes.io/projected/2a7ac8b0-b459-4975-9afc-ed9d9d7a8158-kube-api-access-kxcf4\") pod \"telemeter-client-66b85cb67-z74gv\" (UID: \"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158\") " pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.521856 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.521772 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" Jul 18 01:51:14.654806 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.654572 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66b85cb67-z74gv"] Jul 18 01:51:14.656781 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:51:14.656732 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7ac8b0_b459_4975_9afc_ed9d9d7a8158.slice/crio-3b83ac7ec361dcbabb8d99882e653ffb5846e4b3dee82dad4d5834c068ea829c WatchSource:0}: Error finding container 3b83ac7ec361dcbabb8d99882e653ffb5846e4b3dee82dad4d5834c068ea829c: Status 404 returned error can't find the container with id 3b83ac7ec361dcbabb8d99882e653ffb5846e4b3dee82dad4d5834c068ea829c Jul 18 01:51:14.994942 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:14.994910 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" event={"ID":"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158","Type":"ContainerStarted","Data":"3b83ac7ec361dcbabb8d99882e653ffb5846e4b3dee82dad4d5834c068ea829c"} Jul 18 01:51:17.005260 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:17.005219 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" event={"ID":"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158","Type":"ContainerStarted","Data":"2262049fd4ea78892eb4e9254e7298346dc28e2f2b9d8ee067d70d4414ed9fe8"} Jul 18 01:51:17.005675 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:17.005309 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" event={"ID":"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158","Type":"ContainerStarted","Data":"d56a324850f3a6732bec18fd02c2aafceef8935bc8a8b4ca8115ac8d43fcc84a"} Jul 18 01:51:17.005675 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:17.005329 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" event={"ID":"2a7ac8b0-b459-4975-9afc-ed9d9d7a8158","Type":"ContainerStarted","Data":"1da3a7dcc8668f0dc26ffd173c08a141d73ed16557b775ce7349bd255b41fc40"} Jul 18 01:51:17.029137 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:17.028725 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-66b85cb67-z74gv" podStartSLOduration=1.420384141 podStartE2EDuration="3.0287038s" totalImagesPullingTime="1.608319659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:51:17.026098807 +0000 UTC m=+181.357339790" watchObservedRunningTime="2026-07-18 01:51:17.0287038 +0000 UTC m=+181.359944785" Jul 18 01:51:17.922852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:17.922818 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57c8cdd768-xb7b4"] Jul 18 01:51:17.925492 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:17.925476 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:17.937820 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:17.937790 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c8cdd768-xb7b4"] Jul 18 01:51:18.041980 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.041927 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-serving-cert\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.041980 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.041992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg62l\" (UniqueName: \"kubernetes.io/projected/01346155-0c85-4703-9388-6b316e4588fc-kube-api-access-dg62l\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.042484 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.042045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-trusted-ca-bundle\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.042484 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.042118 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-console-config\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.042484 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.042182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-service-ca\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.042484 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.042208 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-oauth-config\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.042484 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.042252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-oauth-serving-cert\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.143324 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.143273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-service-ca\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.143324 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.143327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-oauth-config\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.143565 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.143362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-oauth-serving-cert\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.143565 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.143400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-serving-cert\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.143565 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.143436 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg62l\" (UniqueName: \"kubernetes.io/projected/01346155-0c85-4703-9388-6b316e4588fc-kube-api-access-dg62l\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.143565 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.143483 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-trusted-ca-bundle\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.143565 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.143528 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-console-config\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.144171 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.144145 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-oauth-serving-cert\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.144258 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.144184 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-service-ca\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.144258 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.144217 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-console-config\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.144340 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.144319 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-trusted-ca-bundle\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.146455 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.146431 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-serving-cert\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.146532 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.146457 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-oauth-config\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.151291 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.151270 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg62l\" (UniqueName: \"kubernetes.io/projected/01346155-0c85-4703-9388-6b316e4588fc-kube-api-access-dg62l\") pod \"console-57c8cdd768-xb7b4\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.234482 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.234384 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:18.365584 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:18.365558 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c8cdd768-xb7b4"] Jul 18 01:51:18.367967 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:51:18.367938 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01346155_0c85_4703_9388_6b316e4588fc.slice/crio-2c29a4d163416b8ab760e1bedcfb948341111068608dfbe9971ae4266bdb382e WatchSource:0}: Error finding container 2c29a4d163416b8ab760e1bedcfb948341111068608dfbe9971ae4266bdb382e: Status 404 returned error can't find the container with id 2c29a4d163416b8ab760e1bedcfb948341111068608dfbe9971ae4266bdb382e Jul 18 01:51:19.014284 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:19.014247 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c8cdd768-xb7b4" event={"ID":"01346155-0c85-4703-9388-6b316e4588fc","Type":"ContainerStarted","Data":"0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9"} Jul 18 01:51:19.014455 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:19.014302 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c8cdd768-xb7b4" event={"ID":"01346155-0c85-4703-9388-6b316e4588fc","Type":"ContainerStarted","Data":"2c29a4d163416b8ab760e1bedcfb948341111068608dfbe9971ae4266bdb382e"} Jul 18 01:51:19.031103 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:19.031051 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57c8cdd768-xb7b4" podStartSLOduration=2.031033259 podStartE2EDuration="2.031033259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:51:19.03091067 +0000 UTC m=+183.362151652" watchObservedRunningTime="2026-07-18 01:51:19.031033259 +0000 UTC m=+183.362274244" Jul 18 01:51:28.235100 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:28.235064 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:28.235513 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:28.235123 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:28.239724 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:28.239702 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:29.027789 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.027754 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57c8cdd768-xb7b4"] Jul 18 01:51:29.058658 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.058626 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54d7bbc549-w7vtc"] Jul 18 01:51:29.060994 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.060975 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:29.061112 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.061078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.070429 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.070409 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d7bbc549-w7vtc"] Jul 18 01:51:29.139617 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.139582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-trusted-ca-bundle\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.139617 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.139619 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-oauth-config\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.139852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.139637 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-service-ca\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.139852 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.139805 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-serving-cert\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.140007 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.139991 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-oauth-serving-cert\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.140061 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.140048 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdl7\" (UniqueName: \"kubernetes.io/projected/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-kube-api-access-qrdl7\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.140103 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.140092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-config\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.241165 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.241127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-oauth-serving-cert\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.241165 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.241173 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdl7\" (UniqueName: \"kubernetes.io/projected/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-kube-api-access-qrdl7\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.241703 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.241197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-config\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.241703 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.241226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-trusted-ca-bundle\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.241703 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.241247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-oauth-config\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.241703 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.241261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-service-ca\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.241703 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.241280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-serving-cert\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.242077 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.242049 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-oauth-serving-cert\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.242077 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.242065 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-service-ca\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.242216 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.242133 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-config\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.242256 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.242210 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-trusted-ca-bundle\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.243910 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.243881 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-oauth-config\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.244001 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.243966 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-serving-cert\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.249138 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.249121 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdl7\" (UniqueName: \"kubernetes.io/projected/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-kube-api-access-qrdl7\") pod \"console-54d7bbc549-w7vtc\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.370278 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.370244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:29.499242 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:29.499213 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d7bbc549-w7vtc"] Jul 18 01:51:29.502159 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:51:29.502132 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bbb3fc_d3a8_43e1_bae7_c40f5ee6e14f.slice/crio-051c9c29333592805441de6346f82a2601053bf4fe5dada2e955a72aa43eb4a0 WatchSource:0}: Error finding container 051c9c29333592805441de6346f82a2601053bf4fe5dada2e955a72aa43eb4a0: Status 404 returned error can't find the container with id 051c9c29333592805441de6346f82a2601053bf4fe5dada2e955a72aa43eb4a0 Jul 18 01:51:30.061016 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:30.060975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d7bbc549-w7vtc" event={"ID":"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f","Type":"ContainerStarted","Data":"b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973"} Jul 18 01:51:30.061204 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:30.061031 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d7bbc549-w7vtc" event={"ID":"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f","Type":"ContainerStarted","Data":"051c9c29333592805441de6346f82a2601053bf4fe5dada2e955a72aa43eb4a0"} Jul 18 01:51:30.091703 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:30.091655 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54d7bbc549-w7vtc" podStartSLOduration=1.09163942 podStartE2EDuration="1.09163942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:51:30.090850469 +0000 UTC m=+194.422091452" watchObservedRunningTime="2026-07-18 01:51:30.09163942 +0000 UTC m=+194.422880401" Jul 18 01:51:39.371437 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:39.371389 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:39.371964 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:39.371466 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:39.376092 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:39.376069 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:40.103031 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:40.103004 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:51:40.152468 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:40.152437 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75985b5c64-mwmvz"] Jul 18 01:51:55.081222 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.081171 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57c8cdd768-xb7b4" podUID="01346155-0c85-4703-9388-6b316e4588fc" containerName="console" containerID="cri-o://0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9" gracePeriod=15 Jul 18 01:51:55.324419 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.324396 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:55.370729 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.370703 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-trusted-ca-bundle\") pod \"01346155-0c85-4703-9388-6b316e4588fc\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " Jul 18 01:51:55.370919 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.370765 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-oauth-config\") pod \"01346155-0c85-4703-9388-6b316e4588fc\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " Jul 18 01:51:55.370919 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.370795 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg62l\" (UniqueName: \"kubernetes.io/projected/01346155-0c85-4703-9388-6b316e4588fc-kube-api-access-dg62l\") pod \"01346155-0c85-4703-9388-6b316e4588fc\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " Jul 18 01:51:55.370919 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.370845 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-oauth-serving-cert\") pod \"01346155-0c85-4703-9388-6b316e4588fc\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " Jul 18 01:51:55.370919 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.370876 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-serving-cert\") pod \"01346155-0c85-4703-9388-6b316e4588fc\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " Jul 18 01:51:55.370919 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.370905 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-service-ca\") pod \"01346155-0c85-4703-9388-6b316e4588fc\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " Jul 18 01:51:55.371160 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.370996 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-console-config\") pod \"01346155-0c85-4703-9388-6b316e4588fc\" (UID: \"01346155-0c85-4703-9388-6b316e4588fc\") " Jul 18 01:51:55.371571 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.371383 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-service-ca" (OuterVolumeSpecName: "service-ca") pod "01346155-0c85-4703-9388-6b316e4588fc" (UID: "01346155-0c85-4703-9388-6b316e4588fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:51:55.371571 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.371334 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01346155-0c85-4703-9388-6b316e4588fc" (UID: "01346155-0c85-4703-9388-6b316e4588fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:51:55.371832 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.371654 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-console-config" (OuterVolumeSpecName: "console-config") pod "01346155-0c85-4703-9388-6b316e4588fc" (UID: "01346155-0c85-4703-9388-6b316e4588fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:51:55.372056 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.372030 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "01346155-0c85-4703-9388-6b316e4588fc" (UID: "01346155-0c85-4703-9388-6b316e4588fc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:51:55.373308 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.373239 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01346155-0c85-4703-9388-6b316e4588fc" (UID: "01346155-0c85-4703-9388-6b316e4588fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:51:55.373447 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.373285 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01346155-0c85-4703-9388-6b316e4588fc" (UID: "01346155-0c85-4703-9388-6b316e4588fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:51:55.373579 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.373553 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01346155-0c85-4703-9388-6b316e4588fc-kube-api-access-dg62l" (OuterVolumeSpecName: "kube-api-access-dg62l") pod "01346155-0c85-4703-9388-6b316e4588fc" (UID: "01346155-0c85-4703-9388-6b316e4588fc"). InnerVolumeSpecName "kube-api-access-dg62l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:51:55.472360 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.472328 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-oauth-serving-cert\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:51:55.472360 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.472354 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-serving-cert\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:51:55.472360 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.472365 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:51:55.472584 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.472374 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-console-config\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:51:55.472584 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.472382 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01346155-0c85-4703-9388-6b316e4588fc-trusted-ca-bundle\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:51:55.472584 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.472391 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01346155-0c85-4703-9388-6b316e4588fc-console-oauth-config\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:51:55.472584 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:55.472399 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg62l\" (UniqueName: \"kubernetes.io/projected/01346155-0c85-4703-9388-6b316e4588fc-kube-api-access-dg62l\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:51:56.160826 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.160797 2582 generic.go:356] "Generic (PLEG): container finished" podID="01346155-0c85-4703-9388-6b316e4588fc" containerID="0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9" exitCode=0 Jul 18 01:51:56.161219 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.160842 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c8cdd768-xb7b4" Jul 18 01:51:56.161219 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.160877 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c8cdd768-xb7b4" event={"ID":"01346155-0c85-4703-9388-6b316e4588fc","Type":"ContainerDied","Data":"0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9"} Jul 18 01:51:56.161219 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.160938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c8cdd768-xb7b4" event={"ID":"01346155-0c85-4703-9388-6b316e4588fc","Type":"ContainerDied","Data":"2c29a4d163416b8ab760e1bedcfb948341111068608dfbe9971ae4266bdb382e"} Jul 18 01:51:56.161219 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.160961 2582 scope.go:117] "RemoveContainer" containerID="0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9" Jul 18 01:51:56.169656 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.169640 2582 scope.go:117] "RemoveContainer" containerID="0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9" Jul 18 01:51:56.169933 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:51:56.169910 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9\": container with ID starting with 0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9 not found: ID does not exist" containerID="0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9" Jul 18 01:51:56.170012 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.169938 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9"} err="failed to get container status \"0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9\": rpc error: code = NotFound desc = could not find container \"0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9\": container with ID starting with 0cdb94415a95157cb9d18e80edcd79ea102bfea4efabefd34313e3596ff76cc9 not found: ID does not exist" Jul 18 01:51:56.180292 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.180270 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57c8cdd768-xb7b4"] Jul 18 01:51:56.186664 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.186640 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57c8cdd768-xb7b4"] Jul 18 01:51:56.193803 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:56.193777 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01346155-0c85-4703-9388-6b316e4588fc" path="/var/lib/kubelet/pods/01346155-0c85-4703-9388-6b316e4588fc/volumes" Jul 18 01:51:59.177418 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.177389 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt"] Jul 18 01:51:59.177807 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.177719 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01346155-0c85-4703-9388-6b316e4588fc" containerName="console" Jul 18 01:51:59.177807 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.177729 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="01346155-0c85-4703-9388-6b316e4588fc" containerName="console" Jul 18 01:51:59.177888 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.177812 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="01346155-0c85-4703-9388-6b316e4588fc" containerName="console" Jul 18 01:51:59.182359 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.182342 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.184966 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.184945 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 18 01:51:59.185813 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.185794 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 18 01:51:59.185907 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.185794 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 18 01:51:59.185907 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.185825 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 18 01:51:59.193026 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.193005 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt"] Jul 18 01:51:59.198461 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.198438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e210433-b1be-4da8-a1fa-e908373ea734-tmp\") pod \"klusterlet-addon-workmgr-5b5769d47-fcltt\" (UID: \"4e210433-b1be-4da8-a1fa-e908373ea734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.198553 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.198497 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4e210433-b1be-4da8-a1fa-e908373ea734-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b5769d47-fcltt\" (UID: \"4e210433-b1be-4da8-a1fa-e908373ea734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.198553 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.198524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt4rs\" (UniqueName: \"kubernetes.io/projected/4e210433-b1be-4da8-a1fa-e908373ea734-kube-api-access-bt4rs\") pod \"klusterlet-addon-workmgr-5b5769d47-fcltt\" (UID: \"4e210433-b1be-4da8-a1fa-e908373ea734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.202432 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.202409 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952"] Jul 18 01:51:59.205943 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.205928 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.209606 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.209589 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 18 01:51:59.209702 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.209624 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 18 01:51:59.209702 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.209591 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 18 01:51:59.209702 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.209591 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 18 01:51:59.221244 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.221223 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952"] Jul 18 01:51:59.299918 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.299882 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-hub\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.299918 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.299920 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.300174 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.299957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtztr\" (UniqueName: \"kubernetes.io/projected/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-kube-api-access-gtztr\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.300174 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.300066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.300174 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.300127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4e210433-b1be-4da8-a1fa-e908373ea734-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b5769d47-fcltt\" (UID: \"4e210433-b1be-4da8-a1fa-e908373ea734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.300174 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.300160 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt4rs\" (UniqueName: \"kubernetes.io/projected/4e210433-b1be-4da8-a1fa-e908373ea734-kube-api-access-bt4rs\") pod \"klusterlet-addon-workmgr-5b5769d47-fcltt\" (UID: \"4e210433-b1be-4da8-a1fa-e908373ea734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.300365 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.300225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.300365 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.300260 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-ca\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.300365 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.300322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e210433-b1be-4da8-a1fa-e908373ea734-tmp\") pod \"klusterlet-addon-workmgr-5b5769d47-fcltt\" (UID: \"4e210433-b1be-4da8-a1fa-e908373ea734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.300728 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.300709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e210433-b1be-4da8-a1fa-e908373ea734-tmp\") pod \"klusterlet-addon-workmgr-5b5769d47-fcltt\" (UID: \"4e210433-b1be-4da8-a1fa-e908373ea734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.302836 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.302818 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4e210433-b1be-4da8-a1fa-e908373ea734-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b5769d47-fcltt\" (UID: \"4e210433-b1be-4da8-a1fa-e908373ea734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.309728 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.309707 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt4rs\" (UniqueName: \"kubernetes.io/projected/4e210433-b1be-4da8-a1fa-e908373ea734-kube-api-access-bt4rs\") pod \"klusterlet-addon-workmgr-5b5769d47-fcltt\" (UID: \"4e210433-b1be-4da8-a1fa-e908373ea734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.401236 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.401201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-hub\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.401236 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.401234 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.401473 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.401256 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtztr\" (UniqueName: \"kubernetes.io/projected/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-kube-api-access-gtztr\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.401473 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.401287 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.401473 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.401349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.401473 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.401393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-ca\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.402070 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.402048 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.403982 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.403956 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.404071 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.404032 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-hub\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.404321 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.404302 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-ca\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.404438 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.404420 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.418071 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.418049 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtztr\" (UniqueName: \"kubernetes.io/projected/4cb874a9-dc56-498d-b0b3-8c99adb1c9c3-kube-api-access-gtztr\") pod \"cluster-proxy-proxy-agent-76d85c4fbb-jw952\" (UID: \"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.492062 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.491972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:51:59.527788 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.527755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" Jul 18 01:51:59.629540 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.629510 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt"] Jul 18 01:51:59.635437 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:51:59.635411 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e210433_b1be_4da8_a1fa_e908373ea734.slice/crio-eafba377fcd1867c51d0eff00b22c7c96d50dbe92d94b42193aca7e22df12584 WatchSource:0}: Error finding container eafba377fcd1867c51d0eff00b22c7c96d50dbe92d94b42193aca7e22df12584: Status 404 returned error can't find the container with id eafba377fcd1867c51d0eff00b22c7c96d50dbe92d94b42193aca7e22df12584 Jul 18 01:51:59.670471 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:51:59.670449 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952"] Jul 18 01:51:59.672603 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:51:59.672577 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cb874a9_dc56_498d_b0b3_8c99adb1c9c3.slice/crio-98e052c3c28f2d93c1b3d146cf6f7cb1530e5e9e37d19e0b4baed8e279d6ceb8 WatchSource:0}: Error finding container 98e052c3c28f2d93c1b3d146cf6f7cb1530e5e9e37d19e0b4baed8e279d6ceb8: Status 404 returned error can't find the container with id 98e052c3c28f2d93c1b3d146cf6f7cb1530e5e9e37d19e0b4baed8e279d6ceb8 Jul 18 01:52:00.178441 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:00.178411 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" event={"ID":"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3","Type":"ContainerStarted","Data":"98e052c3c28f2d93c1b3d146cf6f7cb1530e5e9e37d19e0b4baed8e279d6ceb8"} Jul 18 01:52:00.181257 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:00.180273 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" event={"ID":"4e210433-b1be-4da8-a1fa-e908373ea734","Type":"ContainerStarted","Data":"eafba377fcd1867c51d0eff00b22c7c96d50dbe92d94b42193aca7e22df12584"} Jul 18 01:52:05.174713 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.174651 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75985b5c64-mwmvz" podUID="7d17c69d-4292-4b7e-a14c-03ba72209903" containerName="console" containerID="cri-o://6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1" gracePeriod=15 Jul 18 01:52:05.207064 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.207036 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" event={"ID":"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3","Type":"ContainerStarted","Data":"d422a361c90662e680df312126b92a8e3aeb34e1f044092a9dcd67aadb374e1e"} Jul 18 01:52:05.207202 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.207081 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" event={"ID":"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3","Type":"ContainerStarted","Data":"10eeb9da1700ca363d48d249f5092ca0a9507047bf5534e2c43e521f8e8b4215"} Jul 18 01:52:05.207202 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.207119 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" event={"ID":"4cb874a9-dc56-498d-b0b3-8c99adb1c9c3","Type":"ContainerStarted","Data":"e0d770570dadbc5ffe5fc0cd19c1b3b97fb962efd18fb46a58b68991dd5b70a6"} Jul 18 01:52:05.208546 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.208522 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" event={"ID":"4e210433-b1be-4da8-a1fa-e908373ea734","Type":"ContainerStarted","Data":"2fd7f1c7c8ff1b6339d40375d923902d048a67aee34872d3d1b10ddd513db9da"} Jul 18 01:52:05.208683 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.208668 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:52:05.210366 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.210349 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" Jul 18 01:52:05.225477 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.225439 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d85c4fbb-jw952" podStartSLOduration=1.184177445 podStartE2EDuration="6.225425376s" totalImagesPullingTime="5.041247931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:52:05.224696918 +0000 UTC m=+229.555937902" watchObservedRunningTime="2026-07-18 01:52:05.225425376 +0000 UTC m=+229.556666358" Jul 18 01:52:05.240922 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.240874 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b5769d47-fcltt" podStartSLOduration=1.164586127 podStartE2EDuration="6.240856303s" totalImagesPullingTime="5.076270176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:52:05.240307188 +0000 UTC m=+229.571548166" watchObservedRunningTime="2026-07-18 01:52:05.240856303 +0000 UTC m=+229.572097287" Jul 18 01:52:05.410724 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.410697 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:52:05.460894 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.460817 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-trusted-ca-bundle\") pod \"7d17c69d-4292-4b7e-a14c-03ba72209903\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " Jul 18 01:52:05.461026 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.460916 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-console-config\") pod \"7d17c69d-4292-4b7e-a14c-03ba72209903\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " Jul 18 01:52:05.461026 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.460938 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-oauth-config\") pod \"7d17c69d-4292-4b7e-a14c-03ba72209903\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " Jul 18 01:52:05.461026 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.460966 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-oauth-serving-cert\") pod \"7d17c69d-4292-4b7e-a14c-03ba72209903\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " Jul 18 01:52:05.461026 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.460992 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-service-ca\") pod \"7d17c69d-4292-4b7e-a14c-03ba72209903\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " Jul 18 01:52:05.461026 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.461013 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-serving-cert\") pod \"7d17c69d-4292-4b7e-a14c-03ba72209903\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " Jul 18 01:52:05.461388 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.461047 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54qtc\" (UniqueName: \"kubernetes.io/projected/7d17c69d-4292-4b7e-a14c-03ba72209903-kube-api-access-54qtc\") pod \"7d17c69d-4292-4b7e-a14c-03ba72209903\" (UID: \"7d17c69d-4292-4b7e-a14c-03ba72209903\") " Jul 18 01:52:05.461388 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.461212 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7d17c69d-4292-4b7e-a14c-03ba72209903" (UID: "7d17c69d-4292-4b7e-a14c-03ba72209903"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:52:05.461388 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.461365 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-console-config" (OuterVolumeSpecName: "console-config") pod "7d17c69d-4292-4b7e-a14c-03ba72209903" (UID: "7d17c69d-4292-4b7e-a14c-03ba72209903"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:52:05.461588 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.461562 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7d17c69d-4292-4b7e-a14c-03ba72209903" (UID: "7d17c69d-4292-4b7e-a14c-03ba72209903"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:52:05.461651 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.461622 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-service-ca" (OuterVolumeSpecName: "service-ca") pod "7d17c69d-4292-4b7e-a14c-03ba72209903" (UID: "7d17c69d-4292-4b7e-a14c-03ba72209903"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:52:05.463297 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.463276 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7d17c69d-4292-4b7e-a14c-03ba72209903" (UID: "7d17c69d-4292-4b7e-a14c-03ba72209903"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:52:05.463493 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.463473 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7d17c69d-4292-4b7e-a14c-03ba72209903" (UID: "7d17c69d-4292-4b7e-a14c-03ba72209903"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:52:05.463540 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.463518 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d17c69d-4292-4b7e-a14c-03ba72209903-kube-api-access-54qtc" (OuterVolumeSpecName: "kube-api-access-54qtc") pod "7d17c69d-4292-4b7e-a14c-03ba72209903" (UID: "7d17c69d-4292-4b7e-a14c-03ba72209903"). InnerVolumeSpecName "kube-api-access-54qtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:52:05.562201 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.562167 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54qtc\" (UniqueName: \"kubernetes.io/projected/7d17c69d-4292-4b7e-a14c-03ba72209903-kube-api-access-54qtc\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:52:05.562201 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.562194 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-trusted-ca-bundle\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:52:05.562201 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.562203 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-console-config\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:52:05.562430 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.562214 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-oauth-config\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:52:05.562430 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.562222 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-oauth-serving-cert\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:52:05.562430 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.562233 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d17c69d-4292-4b7e-a14c-03ba72209903-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:52:05.562430 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:05.562241 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d17c69d-4292-4b7e-a14c-03ba72209903-console-serving-cert\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:52:06.213573 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:06.213547 2582 generic.go:356] "Generic (PLEG): container finished" podID="7d17c69d-4292-4b7e-a14c-03ba72209903" containerID="6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1" exitCode=0 Jul 18 01:52:06.213974 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:06.213587 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75985b5c64-mwmvz" Jul 18 01:52:06.213974 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:06.213619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75985b5c64-mwmvz" event={"ID":"7d17c69d-4292-4b7e-a14c-03ba72209903","Type":"ContainerDied","Data":"6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1"} Jul 18 01:52:06.213974 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:06.213664 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75985b5c64-mwmvz" event={"ID":"7d17c69d-4292-4b7e-a14c-03ba72209903","Type":"ContainerDied","Data":"682e109353b1e228800b4a15569b6abd077c2f373f613049749509a2fbd91a82"} Jul 18 01:52:06.213974 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:06.213692 2582 scope.go:117] "RemoveContainer" containerID="6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1" Jul 18 01:52:06.222167 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:06.222110 2582 scope.go:117] "RemoveContainer" containerID="6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1" Jul 18 01:52:06.222390 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:52:06.222367 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1\": container with ID starting with 6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1 not found: ID does not exist" containerID="6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1" Jul 18 01:52:06.222451 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:06.222399 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1"} err="failed to get container status \"6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1\": rpc error: code = NotFound desc = could not find container \"6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1\": container with ID starting with 6bc2426ca25230443d3de92e1446dd5f5f55f46c5a1538900056b9775dccdcf1 not found: ID does not exist" Jul 18 01:52:06.230719 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:06.230692 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75985b5c64-mwmvz"] Jul 18 01:52:06.236942 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:06.236919 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75985b5c64-mwmvz"] Jul 18 01:52:08.193921 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:52:08.193887 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d17c69d-4292-4b7e-a14c-03ba72209903" path="/var/lib/kubelet/pods/7d17c69d-4292-4b7e-a14c-03ba72209903/volumes" Jul 18 01:53:10.872829 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.872792 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4"] Jul 18 01:53:10.873265 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.873085 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d17c69d-4292-4b7e-a14c-03ba72209903" containerName="console" Jul 18 01:53:10.873265 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.873095 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d17c69d-4292-4b7e-a14c-03ba72209903" containerName="console" Jul 18 01:53:10.873265 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.873162 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d17c69d-4292-4b7e-a14c-03ba72209903" containerName="console" Jul 18 01:53:10.875100 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.875085 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" Jul 18 01:53:10.877353 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.877329 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 18 01:53:10.877474 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.877375 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 18 01:53:10.878137 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.878114 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ssllj\"" Jul 18 01:53:10.883601 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.883576 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4"] Jul 18 01:53:10.995098 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.995055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b933d8b2-77d2-40f0-81aa-7cc229fd2599-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2vsv4\" (UID: \"b933d8b2-77d2-40f0-81aa-7cc229fd2599\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" Jul 18 01:53:10.995304 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:10.995122 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9wg5\" (UniqueName: \"kubernetes.io/projected/b933d8b2-77d2-40f0-81aa-7cc229fd2599-kube-api-access-v9wg5\") pod \"openshift-lws-operator-fd8ccff4c-2vsv4\" (UID: \"b933d8b2-77d2-40f0-81aa-7cc229fd2599\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" Jul 18 01:53:11.095705 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:11.095654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9wg5\" (UniqueName: \"kubernetes.io/projected/b933d8b2-77d2-40f0-81aa-7cc229fd2599-kube-api-access-v9wg5\") pod \"openshift-lws-operator-fd8ccff4c-2vsv4\" (UID: \"b933d8b2-77d2-40f0-81aa-7cc229fd2599\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" Jul 18 01:53:11.095919 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:11.095811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b933d8b2-77d2-40f0-81aa-7cc229fd2599-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2vsv4\" (UID: \"b933d8b2-77d2-40f0-81aa-7cc229fd2599\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" Jul 18 01:53:11.096161 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:11.096142 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b933d8b2-77d2-40f0-81aa-7cc229fd2599-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2vsv4\" (UID: \"b933d8b2-77d2-40f0-81aa-7cc229fd2599\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" Jul 18 01:53:11.103425 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:11.103400 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9wg5\" (UniqueName: \"kubernetes.io/projected/b933d8b2-77d2-40f0-81aa-7cc229fd2599-kube-api-access-v9wg5\") pod \"openshift-lws-operator-fd8ccff4c-2vsv4\" (UID: \"b933d8b2-77d2-40f0-81aa-7cc229fd2599\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" Jul 18 01:53:11.184837 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:11.184761 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" Jul 18 01:53:11.314587 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:11.314556 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4"] Jul 18 01:53:11.317676 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:53:11.317649 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb933d8b2_77d2_40f0_81aa_7cc229fd2599.slice/crio-d146ddb16fa7bcd4d50b5e6fc6c9bee38633f011ad0bd2c1c6aed3e8a312d14a WatchSource:0}: Error finding container d146ddb16fa7bcd4d50b5e6fc6c9bee38633f011ad0bd2c1c6aed3e8a312d14a: Status 404 returned error can't find the container with id d146ddb16fa7bcd4d50b5e6fc6c9bee38633f011ad0bd2c1c6aed3e8a312d14a Jul 18 01:53:11.473271 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:11.473186 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" event={"ID":"b933d8b2-77d2-40f0-81aa-7cc229fd2599","Type":"ContainerStarted","Data":"d146ddb16fa7bcd4d50b5e6fc6c9bee38633f011ad0bd2c1c6aed3e8a312d14a"} Jul 18 01:53:14.488491 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:14.488451 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" event={"ID":"b933d8b2-77d2-40f0-81aa-7cc229fd2599","Type":"ContainerStarted","Data":"dbe8934b3c17c98e2d8f9d01757c51505779dc8d231b9d2afc4f7cdd82c9841e"} Jul 18 01:53:14.503521 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:14.503463 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2vsv4" podStartSLOduration=2.113084686 podStartE2EDuration="4.503443367s" totalImagesPullingTime="2.390358681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:53:14.50299999 +0000 UTC m=+298.834240973" watchObservedRunningTime="2026-07-18 01:53:14.503443367 +0000 UTC m=+298.834684350" Jul 18 01:53:16.080591 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:16.080566 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 18 01:53:30.718119 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.718081 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp"] Jul 18 01:53:30.724307 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.724287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.728155 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.728123 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 18 01:53:30.730947 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.730923 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8n9fz\"" Jul 18 01:53:30.732191 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.732173 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 18 01:53:30.732284 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.732178 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 18 01:53:30.732583 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.732560 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 18 01:53:30.757475 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.757449 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp"] Jul 18 01:53:30.864102 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.864057 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/740e4863-0fa0-459a-a89e-1fe74ec76b16-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.864276 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.864118 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/740e4863-0fa0-459a-a89e-1fe74ec76b16-manifests\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.864276 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.864135 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/740e4863-0fa0-459a-a89e-1fe74ec76b16-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.864276 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.864157 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw2ph\" (UniqueName: \"kubernetes.io/projected/740e4863-0fa0-459a-a89e-1fe74ec76b16-kube-api-access-xw2ph\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.864276 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.864185 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740e4863-0fa0-459a-a89e-1fe74ec76b16-tmp\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.965388 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.965351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/740e4863-0fa0-459a-a89e-1fe74ec76b16-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.965569 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.965409 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/740e4863-0fa0-459a-a89e-1fe74ec76b16-manifests\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.965569 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.965437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/740e4863-0fa0-459a-a89e-1fe74ec76b16-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.965569 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.965460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw2ph\" (UniqueName: \"kubernetes.io/projected/740e4863-0fa0-459a-a89e-1fe74ec76b16-kube-api-access-xw2ph\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.965569 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.965497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740e4863-0fa0-459a-a89e-1fe74ec76b16-tmp\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.965950 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.965927 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/740e4863-0fa0-459a-a89e-1fe74ec76b16-manifests\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.968061 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.968036 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740e4863-0fa0-459a-a89e-1fe74ec76b16-tmp\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.968188 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.968150 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/740e4863-0fa0-459a-a89e-1fe74ec76b16-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.968273 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.968258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/740e4863-0fa0-459a-a89e-1fe74ec76b16-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:30.973384 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:30.973366 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw2ph\" (UniqueName: \"kubernetes.io/projected/740e4863-0fa0-459a-a89e-1fe74ec76b16-kube-api-access-xw2ph\") pod \"opendatahub-operator-controller-manager-6b7456dd48-59hvp\" (UID: \"740e4863-0fa0-459a-a89e-1fe74ec76b16\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:31.034777 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:31.034724 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:31.194227 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:31.194200 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp"] Jul 18 01:53:31.197243 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:53:31.197218 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740e4863_0fa0_459a_a89e_1fe74ec76b16.slice/crio-e754302fc62d56453a36d61a5a8b457d556890de554dd905b9c3c8c3a1f48ff6 WatchSource:0}: Error finding container e754302fc62d56453a36d61a5a8b457d556890de554dd905b9c3c8c3a1f48ff6: Status 404 returned error can't find the container with id e754302fc62d56453a36d61a5a8b457d556890de554dd905b9c3c8c3a1f48ff6 Jul 18 01:53:31.199696 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:31.199678 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 01:53:31.557071 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:31.557040 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" event={"ID":"740e4863-0fa0-459a-a89e-1fe74ec76b16","Type":"ContainerStarted","Data":"e754302fc62d56453a36d61a5a8b457d556890de554dd905b9c3c8c3a1f48ff6"} Jul 18 01:53:34.574598 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:34.574505 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" event={"ID":"740e4863-0fa0-459a-a89e-1fe74ec76b16","Type":"ContainerStarted","Data":"bcb6c5b440e78f2d69c43bb3fc1d3902f33f1a468924b0de202d1a60a94ad19e"} Jul 18 01:53:35.579929 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:35.579837 2582 generic.go:356] "Generic (PLEG): container finished" podID="740e4863-0fa0-459a-a89e-1fe74ec76b16" containerID="bcb6c5b440e78f2d69c43bb3fc1d3902f33f1a468924b0de202d1a60a94ad19e" exitCode=0 Jul 18 01:53:35.579929 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:35.579900 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" event={"ID":"740e4863-0fa0-459a-a89e-1fe74ec76b16","Type":"ContainerDied","Data":"bcb6c5b440e78f2d69c43bb3fc1d3902f33f1a468924b0de202d1a60a94ad19e"} Jul 18 01:53:36.588750 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:36.588652 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" event={"ID":"740e4863-0fa0-459a-a89e-1fe74ec76b16","Type":"ContainerStarted","Data":"debded9444d112b1cd8d4e8fb6984b09bd87ef9ba40c26c7dfe6e8aa6700de74"} Jul 18 01:53:36.589114 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:36.588851 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:53:36.611053 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:36.611006 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" podStartSLOduration=2.009703993 podStartE2EDuration="6.61099003s" totalImagesPullingTime="3.505254169s" totalInitContainerRuntime="1.096031868s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:53:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:53:36.609378479 +0000 UTC m=+320.940619464" watchObservedRunningTime="2026-07-18 01:53:36.61099003 +0000 UTC m=+320.942231011" Jul 18 01:53:47.596399 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:53:47.596368 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b7456dd48-59hvp" Jul 18 01:54:00.084130 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.084094 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d"] Jul 18 01:54:00.091662 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.091631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.096552 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.096525 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 18 01:54:00.096701 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.096560 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 18 01:54:00.096859 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.096844 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 18 01:54:00.097511 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.097493 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 18 01:54:00.097867 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.097853 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ctngl\"" Jul 18 01:54:00.101551 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.101528 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 18 01:54:00.101649 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.101579 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 18 01:54:00.103205 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.103182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f69470a-71fe-4730-8bac-d716db010398-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.103290 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.103232 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f69470a-71fe-4730-8bac-d716db010398-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.103290 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.103258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f69470a-71fe-4730-8bac-d716db010398-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.103399 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.103296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fdjv\" (UniqueName: \"kubernetes.io/projected/0f69470a-71fe-4730-8bac-d716db010398-kube-api-access-7fdjv\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.103399 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.103360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f69470a-71fe-4730-8bac-d716db010398-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.103399 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.103384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f69470a-71fe-4730-8bac-d716db010398-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.103508 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.103438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f69470a-71fe-4730-8bac-d716db010398-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.116960 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.116936 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d"] Jul 18 01:54:00.204816 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.204779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f69470a-71fe-4730-8bac-d716db010398-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.204816 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.204812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f69470a-71fe-4730-8bac-d716db010398-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.205079 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.204837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f69470a-71fe-4730-8bac-d716db010398-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.205079 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.204858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f69470a-71fe-4730-8bac-d716db010398-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.205079 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.204889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f69470a-71fe-4730-8bac-d716db010398-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.205079 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.204905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f69470a-71fe-4730-8bac-d716db010398-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.205079 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.204924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fdjv\" (UniqueName: \"kubernetes.io/projected/0f69470a-71fe-4730-8bac-d716db010398-kube-api-access-7fdjv\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.205715 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.205694 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f69470a-71fe-4730-8bac-d716db010398-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.207836 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.207816 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f69470a-71fe-4730-8bac-d716db010398-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.207927 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.207906 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f69470a-71fe-4730-8bac-d716db010398-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.207987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.207928 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f69470a-71fe-4730-8bac-d716db010398-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.208040 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.207991 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f69470a-71fe-4730-8bac-d716db010398-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.230166 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.230138 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fdjv\" (UniqueName: \"kubernetes.io/projected/0f69470a-71fe-4730-8bac-d716db010398-kube-api-access-7fdjv\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.230308 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.230227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f69470a-71fe-4730-8bac-d716db010398-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-55w5d\" (UID: \"0f69470a-71fe-4730-8bac-d716db010398\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.401012 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.400719 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:00.559703 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.559678 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d"] Jul 18 01:54:00.560973 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:54:00.560952 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f69470a_71fe_4730_8bac_d716db010398.slice/crio-aafc32bde2a3e4e2fc30fa72ccedfdef02d36a313e5de73f50adb3beba302460 WatchSource:0}: Error finding container aafc32bde2a3e4e2fc30fa72ccedfdef02d36a313e5de73f50adb3beba302460: Status 404 returned error can't find the container with id aafc32bde2a3e4e2fc30fa72ccedfdef02d36a313e5de73f50adb3beba302460 Jul 18 01:54:00.591960 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.591932 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj"] Jul 18 01:54:00.596242 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.596223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.598915 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.598892 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 18 01:54:00.599017 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.598963 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 18 01:54:00.599122 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.599110 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9t7v9\"" Jul 18 01:54:00.599850 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.599836 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 18 01:54:00.610242 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.610220 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj"] Jul 18 01:54:00.610802 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.610782 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/51e9b6eb-cbbe-469f-b702-d26915483616-manager-config\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.610911 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.610818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/51e9b6eb-cbbe-469f-b702-d26915483616-metrics-cert\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.610911 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.610838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmsb\" (UniqueName: \"kubernetes.io/projected/51e9b6eb-cbbe-469f-b702-d26915483616-kube-api-access-5zmsb\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.611016 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.610928 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51e9b6eb-cbbe-469f-b702-d26915483616-cert\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.704730 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.704649 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" event={"ID":"0f69470a-71fe-4730-8bac-d716db010398","Type":"ContainerStarted","Data":"aafc32bde2a3e4e2fc30fa72ccedfdef02d36a313e5de73f50adb3beba302460"} Jul 18 01:54:00.711367 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.711345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/51e9b6eb-cbbe-469f-b702-d26915483616-manager-config\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.711486 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.711376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/51e9b6eb-cbbe-469f-b702-d26915483616-metrics-cert\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.711486 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.711392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmsb\" (UniqueName: \"kubernetes.io/projected/51e9b6eb-cbbe-469f-b702-d26915483616-kube-api-access-5zmsb\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.711486 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.711420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51e9b6eb-cbbe-469f-b702-d26915483616-cert\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.712035 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.712014 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/51e9b6eb-cbbe-469f-b702-d26915483616-manager-config\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.713972 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.713955 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51e9b6eb-cbbe-469f-b702-d26915483616-cert\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.714043 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.713992 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/51e9b6eb-cbbe-469f-b702-d26915483616-metrics-cert\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.726146 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.726126 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmsb\" (UniqueName: \"kubernetes.io/projected/51e9b6eb-cbbe-469f-b702-d26915483616-kube-api-access-5zmsb\") pod \"lws-controller-manager-6c55bb847-xtnpj\" (UID: \"51e9b6eb-cbbe-469f-b702-d26915483616\") " pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:00.906650 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:00.906617 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:01.045607 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:01.045526 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj"] Jul 18 01:54:01.052241 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:54:01.052198 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e9b6eb_cbbe_469f_b702_d26915483616.slice/crio-49d33513a8408939b6db2d0dc9b2f1add65c91cb6933aaf805a3247718c168fd WatchSource:0}: Error finding container 49d33513a8408939b6db2d0dc9b2f1add65c91cb6933aaf805a3247718c168fd: Status 404 returned error can't find the container with id 49d33513a8408939b6db2d0dc9b2f1add65c91cb6933aaf805a3247718c168fd Jul 18 01:54:01.710847 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:01.710803 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" event={"ID":"51e9b6eb-cbbe-469f-b702-d26915483616","Type":"ContainerStarted","Data":"49d33513a8408939b6db2d0dc9b2f1add65c91cb6933aaf805a3247718c168fd"} Jul 18 01:54:03.433953 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:03.433910 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 18 01:54:03.434231 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:03.433990 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 18 01:54:03.723514 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:03.723418 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" event={"ID":"51e9b6eb-cbbe-469f-b702-d26915483616","Type":"ContainerStarted","Data":"f08077029ef20fda8acb7c4f491830d78467e14f886c9ce0b77e8b6fd587af05"} Jul 18 01:54:03.723514 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:03.723511 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:54:03.725128 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:03.725100 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" event={"ID":"0f69470a-71fe-4730-8bac-d716db010398","Type":"ContainerStarted","Data":"425ddee311a83bb2b8211c94935a4fd1449c6c3378364c05321590c54515448a"} Jul 18 01:54:03.725255 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:03.725151 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:03.744296 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:03.744256 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" podStartSLOduration=1.364301953 podStartE2EDuration="3.744243384s" totalImagesPullingTime="2.379941431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:54:03.740953117 +0000 UTC m=+348.072194098" watchObservedRunningTime="2026-07-18 01:54:03.744243384 +0000 UTC m=+348.075484430" Jul 18 01:54:03.766831 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:03.766792 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" podStartSLOduration=0.896121245 podStartE2EDuration="3.766776599s" totalImagesPullingTime="2.870655354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:54:03.765485734 +0000 UTC m=+348.096726716" watchObservedRunningTime="2026-07-18 01:54:03.766776599 +0000 UTC m=+348.098017581" Jul 18 01:54:04.730795 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:04.730758 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-55w5d" Jul 18 01:54:14.731820 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:54:14.731726 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c55bb847-xtnpj" Jul 18 01:56:24.976104 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:24.976074 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:56:24.979456 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:24.979434 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:56:24.981979 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:24.981954 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 18 01:56:24.982432 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:24.982410 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:56:24.982886 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:24.982862 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 18 01:56:24.982986 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:24.982911 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qp2sp\"" Jul 18 01:56:24.983233 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:24.982870 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 18 01:56:25.010259 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:25.010230 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:56:25.074722 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:25.074690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj84l\" (UniqueName: \"kubernetes.io/projected/e38b18d2-a337-4407-b215-a47a5ac90eed-kube-api-access-lj84l\") pod \"limitador-limitador-54d6d988d5-jrznl\" (UID: \"e38b18d2-a337-4407-b215-a47a5ac90eed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:56:25.074936 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:25.074777 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e38b18d2-a337-4407-b215-a47a5ac90eed-config-file\") pod \"limitador-limitador-54d6d988d5-jrznl\" (UID: \"e38b18d2-a337-4407-b215-a47a5ac90eed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:56:25.175905 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:25.175867 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e38b18d2-a337-4407-b215-a47a5ac90eed-config-file\") pod \"limitador-limitador-54d6d988d5-jrznl\" (UID: \"e38b18d2-a337-4407-b215-a47a5ac90eed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:56:25.176091 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:25.175936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj84l\" (UniqueName: \"kubernetes.io/projected/e38b18d2-a337-4407-b215-a47a5ac90eed-kube-api-access-lj84l\") pod \"limitador-limitador-54d6d988d5-jrznl\" (UID: \"e38b18d2-a337-4407-b215-a47a5ac90eed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:56:25.176577 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:25.176551 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e38b18d2-a337-4407-b215-a47a5ac90eed-config-file\") pod \"limitador-limitador-54d6d988d5-jrznl\" (UID: \"e38b18d2-a337-4407-b215-a47a5ac90eed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:56:25.187196 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:25.187162 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj84l\" (UniqueName: \"kubernetes.io/projected/e38b18d2-a337-4407-b215-a47a5ac90eed-kube-api-access-lj84l\") pod \"limitador-limitador-54d6d988d5-jrznl\" (UID: \"e38b18d2-a337-4407-b215-a47a5ac90eed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:56:25.292623 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:25.292539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:56:25.440318 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:25.440282 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:56:25.443582 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:56:25.443555 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38b18d2_a337_4407_b215_a47a5ac90eed.slice/crio-3cb3f75afcd56cbaaad8c49fe56a03d940b5ee54c2823d4c505aafb3e8dbb59d WatchSource:0}: Error finding container 3cb3f75afcd56cbaaad8c49fe56a03d940b5ee54c2823d4c505aafb3e8dbb59d: Status 404 returned error can't find the container with id 3cb3f75afcd56cbaaad8c49fe56a03d940b5ee54c2823d4c505aafb3e8dbb59d Jul 18 01:56:26.359264 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:26.359225 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" event={"ID":"e38b18d2-a337-4407-b215-a47a5ac90eed","Type":"ContainerStarted","Data":"3cb3f75afcd56cbaaad8c49fe56a03d940b5ee54c2823d4c505aafb3e8dbb59d"} Jul 18 01:56:28.370690 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:28.370654 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" event={"ID":"e38b18d2-a337-4407-b215-a47a5ac90eed","Type":"ContainerStarted","Data":"afc667ce65787f5f7dc1de2a0f383541e65f8b29efc594f5778c818e421c67c3"} Jul 18 01:56:28.371084 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:28.370782 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:56:28.389581 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:28.389532 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" podStartSLOduration=1.9112281420000001 podStartE2EDuration="4.389512783s" totalImagesPullingTime="2.478284641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:56:28.388437188 +0000 UTC m=+492.719678171" watchObservedRunningTime="2026-07-18 01:56:28.389512783 +0000 UTC m=+492.720753767" Jul 18 01:56:35.931595 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:35.931553 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d7bbc549-w7vtc"] Jul 18 01:56:39.376953 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:56:39.376926 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrznl" Jul 18 01:57:00.953205 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:00.953162 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54d7bbc549-w7vtc" podUID="11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" containerName="console" containerID="cri-o://b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973" gracePeriod=15 Jul 18 01:57:01.194996 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.194964 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:57:01.208081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.208009 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-oauth-serving-cert\") pod \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " Jul 18 01:57:01.208081 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.208042 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-serving-cert\") pod \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " Jul 18 01:57:01.208337 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.208316 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" (UID: "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:57:01.210208 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.210178 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" (UID: "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:57:01.308918 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.308878 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-trusted-ca-bundle\") pod \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " Jul 18 01:57:01.308918 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.308926 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-config\") pod \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " Jul 18 01:57:01.309161 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.308967 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-service-ca\") pod \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " Jul 18 01:57:01.309161 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.308986 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-oauth-config\") pod \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " Jul 18 01:57:01.309161 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.309003 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrdl7\" (UniqueName: \"kubernetes.io/projected/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-kube-api-access-qrdl7\") pod \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\" (UID: \"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f\") " Jul 18 01:57:01.309319 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.309238 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-oauth-serving-cert\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:57:01.309319 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.309255 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-serving-cert\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:57:01.309415 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.309369 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" (UID: "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:57:01.309415 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.309392 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-service-ca" (OuterVolumeSpecName: "service-ca") pod "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" (UID: "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:57:01.309415 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.309400 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-config" (OuterVolumeSpecName: "console-config") pod "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" (UID: "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:57:01.311316 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.311291 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" (UID: "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:57:01.311418 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.311341 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-kube-api-access-qrdl7" (OuterVolumeSpecName: "kube-api-access-qrdl7") pod "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" (UID: "11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f"). InnerVolumeSpecName "kube-api-access-qrdl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:57:01.409860 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.409824 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-trusted-ca-bundle\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:57:01.409860 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.409856 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-config\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:57:01.410052 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.409871 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:57:01.410052 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.409881 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-console-oauth-config\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:57:01.410052 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.409891 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrdl7\" (UniqueName: \"kubernetes.io/projected/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f-kube-api-access-qrdl7\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 01:57:01.518461 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.518370 2582 generic.go:356] "Generic (PLEG): container finished" podID="11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" containerID="b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973" exitCode=0 Jul 18 01:57:01.518461 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.518416 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d7bbc549-w7vtc" event={"ID":"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f","Type":"ContainerDied","Data":"b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973"} Jul 18 01:57:01.518461 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.518436 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d7bbc549-w7vtc" Jul 18 01:57:01.518715 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.518479 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d7bbc549-w7vtc" event={"ID":"11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f","Type":"ContainerDied","Data":"051c9c29333592805441de6346f82a2601053bf4fe5dada2e955a72aa43eb4a0"} Jul 18 01:57:01.518715 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.518499 2582 scope.go:117] "RemoveContainer" containerID="b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973" Jul 18 01:57:01.527139 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.527128 2582 scope.go:117] "RemoveContainer" containerID="b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973" Jul 18 01:57:01.527400 ip-10-0-135-157 kubenswrapper[2582]: E0718 01:57:01.527380 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973\": container with ID starting with b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973 not found: ID does not exist" containerID="b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973" Jul 18 01:57:01.527480 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.527412 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973"} err="failed to get container status \"b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973\": rpc error: code = NotFound desc = could not find container \"b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973\": container with ID starting with b47490832d9c9a1f3cf7e471db0a4a65c72af040bd5cf87c037cef94b9a40973 not found: ID does not exist" Jul 18 01:57:01.539186 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.539148 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d7bbc549-w7vtc"] Jul 18 01:57:01.544233 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:01.544206 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54d7bbc549-w7vtc"] Jul 18 01:57:02.195698 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:02.195661 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" path="/var/lib/kubelet/pods/11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f/volumes" Jul 18 01:57:38.137984 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:57:38.137952 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:58:10.497134 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:10.497092 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:58:20.198267 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:20.197616 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:58:23.960008 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:23.959973 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:58:29.863987 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:29.863953 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:58:34.744790 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.744753 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6"] Jul 18 01:58:34.749006 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.745258 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" containerName="console" Jul 18 01:58:34.749006 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.745274 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" containerName="console" Jul 18 01:58:34.749006 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.745373 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bbb3fc-d3a8-43e1-bae7-c40f5ee6e14f" containerName="console" Jul 18 01:58:34.750587 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.750564 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.754079 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.754060 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tfrql\"" Jul 18 01:58:34.754209 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.754099 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 18 01:58:34.754209 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.754110 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 18 01:58:34.754209 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.754099 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 18 01:58:34.758100 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.757914 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6"] Jul 18 01:58:34.836795 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.836731 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.836995 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.836811 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.836995 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.836844 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.836995 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.836888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.836995 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.836916 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9jj\" (UniqueName: \"kubernetes.io/projected/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-kube-api-access-zp9jj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.837171 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.837007 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.938321 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.938289 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.938321 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.938326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.938576 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.938345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.938576 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.938363 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.938576 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.938386 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9jj\" (UniqueName: \"kubernetes.io/projected/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-kube-api-access-zp9jj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.938576 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.938452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.938844 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.938823 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.938897 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.938864 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.938897 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.938881 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.940785 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.940762 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.940890 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.940874 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:34.948392 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:34.948358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9jj\" (UniqueName: \"kubernetes.io/projected/9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb-kube-api-access-zp9jj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6cvz6\" (UID: \"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:35.062488 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:35.062388 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:35.196429 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:35.196402 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6"] Jul 18 01:58:35.198495 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:58:35.198470 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d30c8f9_fe02_46dc_9f96_ccffd94d9cbb.slice/crio-54e6435320ee2cca573307bb668e778391fe90d726395ad301fb819d0689dd04 WatchSource:0}: Error finding container 54e6435320ee2cca573307bb668e778391fe90d726395ad301fb819d0689dd04: Status 404 returned error can't find the container with id 54e6435320ee2cca573307bb668e778391fe90d726395ad301fb819d0689dd04 Jul 18 01:58:35.199702 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:35.199684 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 01:58:35.932347 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:35.932259 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" event={"ID":"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb","Type":"ContainerStarted","Data":"54e6435320ee2cca573307bb668e778391fe90d726395ad301fb819d0689dd04"} Jul 18 01:58:36.938109 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:36.938071 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" event={"ID":"9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb","Type":"ContainerStarted","Data":"94dc9f1a0df3bb9360199dda2e8c09d20edc969f20b4e552e531eefd0da4d170"} Jul 18 01:58:36.957657 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:36.957606 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" podStartSLOduration=1.937805291 podStartE2EDuration="2.957588165s" totalImagesPullingTime="1.019782874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:58:36.955710401 +0000 UTC m=+621.286951407" watchObservedRunningTime="2026-07-18 01:58:36.957588165 +0000 UTC m=+621.288829125" Jul 18 01:58:37.432544 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:37.432510 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:58:45.063285 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:45.063244 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:45.063815 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:45.063316 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:45.075876 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:45.075852 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:45.988688 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:45.988662 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6cvz6" Jul 18 01:58:51.669710 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.669678 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm"] Jul 18 01:58:51.673060 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.673042 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.675446 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.675422 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 18 01:58:51.683158 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.683137 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm"] Jul 18 01:58:51.789241 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.789212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec05950-dffb-40c7-a4de-0bedade71d44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.789413 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.789252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.789413 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.789327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.789413 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.789372 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.789413 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.789392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czcfv\" (UniqueName: \"kubernetes.io/projected/aec05950-dffb-40c7-a4de-0bedade71d44-kube-api-access-czcfv\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.789544 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.789455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.890520 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.890486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.890693 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.890533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec05950-dffb-40c7-a4de-0bedade71d44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.890693 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.890576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.890693 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.890636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.890693 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.890658 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.890958 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.890815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czcfv\" (UniqueName: \"kubernetes.io/projected/aec05950-dffb-40c7-a4de-0bedade71d44-kube-api-access-czcfv\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.891023 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.890995 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.891023 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.891010 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.891119 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.891096 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.893013 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.892991 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec05950-dffb-40c7-a4de-0bedade71d44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.893282 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.893260 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec05950-dffb-40c7-a4de-0bedade71d44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.898593 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.898567 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czcfv\" (UniqueName: \"kubernetes.io/projected/aec05950-dffb-40c7-a4de-0bedade71d44-kube-api-access-czcfv\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm\" (UID: \"aec05950-dffb-40c7-a4de-0bedade71d44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:51.982979 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:51.982891 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:58:52.112353 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:52.112329 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm"] Jul 18 01:58:52.114628 ip-10-0-135-157 kubenswrapper[2582]: W0718 01:58:52.114599 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaec05950_dffb_40c7_a4de_0bedade71d44.slice/crio-d8375412da189f6b10c521233643160e7d8e606c35f07fc834c4709efa78c8c7 WatchSource:0}: Error finding container d8375412da189f6b10c521233643160e7d8e606c35f07fc834c4709efa78c8c7: Status 404 returned error can't find the container with id d8375412da189f6b10c521233643160e7d8e606c35f07fc834c4709efa78c8c7 Jul 18 01:58:53.011320 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:53.011286 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" event={"ID":"aec05950-dffb-40c7-a4de-0bedade71d44","Type":"ContainerStarted","Data":"c98582e0e5bcd61bfa33df0f9b18349774385b34d41a53c087c095a812207351"} Jul 18 01:58:53.011320 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:53.011324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" event={"ID":"aec05950-dffb-40c7-a4de-0bedade71d44","Type":"ContainerStarted","Data":"d8375412da189f6b10c521233643160e7d8e606c35f07fc834c4709efa78c8c7"} Jul 18 01:58:53.028072 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:53.028023 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" podStartSLOduration=1.8750536279999999 podStartE2EDuration="2.028006838s" totalImagesPullingTime="152.95321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:58:53.027053273 +0000 UTC m=+637.358294250" watchObservedRunningTime="2026-07-18 01:58:53.028006838 +0000 UTC m=+637.359247823" Jul 18 01:58:54.464871 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:58:54.464840 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 01:59:01.983498 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:59:01.983446 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:59:01.983972 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:59:01.983515 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:59:01.995749 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:59:01.995710 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 01:59:02.064046 ip-10-0-135-157 kubenswrapper[2582]: I0718 01:59:02.064016 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm" Jul 18 02:00:34.674064 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:00:34.674034 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:00:43.962149 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:00:43.962106 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:00:54.471490 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:00:54.471456 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:02:37.571188 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:02:37.571153 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:02:46.057087 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:02:46.057054 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:03:19.459714 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:03:19.459673 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:03:32.174123 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:03:32.174088 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:03:45.061219 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:03:45.061181 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:03:52.863323 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:03:52.863287 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:04:03.864226 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:04:03.864183 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:04:13.563022 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:04:13.562974 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:04:24.458047 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:04:24.458013 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:04:34.258546 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:04:34.258508 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:05:00.064623 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:05:00.064539 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:05:11.668732 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:05:11.668696 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:05:34.762527 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:05:34.762493 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:05:45.668382 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:05:45.668346 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:06:00.466576 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:06:00.466539 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:06:10.737332 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:06:10.737244 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:06:20.562890 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:06:20.562852 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:06:30.667899 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:06:30.667862 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:06:40.167431 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:06:40.167398 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:08:28.566294 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:08:28.566258 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:08:40.262444 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:08:40.262401 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:08:49.272669 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:08:49.272637 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:09:03.467274 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:09:03.467239 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:09:12.766032 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:09:12.765946 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:09:21.765971 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:09:21.765936 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:09:31.859083 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:09:31.859050 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:09:42.657239 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:09:42.657201 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:09:53.664685 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:09:53.664650 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:10:02.456246 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:10:02.456214 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:10:12.066780 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:10:12.066726 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:10:35.331760 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:10:35.331702 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:10:43.975542 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:10:43.975460 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:10:54.064968 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:10:54.064927 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:11:05.361540 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:11:05.361509 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:11:22.069835 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:11:22.069796 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:11:31.165292 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:11:31.165263 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:11:40.865853 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:11:40.865814 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:11:51.362873 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:11:51.362838 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:12:00.177856 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:12:00.177824 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:12:10.259631 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:12:10.259547 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:12:24.066161 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:12:24.066124 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:12:33.464667 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:12:33.464629 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:12:42.671165 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:12:42.671132 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:12:55.064535 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:12:55.064505 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:13:07.375982 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:13:07.375947 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:13:15.869375 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:13:15.869343 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:13:28.360857 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:13:28.360823 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:13:39.965879 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:13:39.965803 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:13:49.571317 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:13:49.571283 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:13:58.475768 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:13:58.475712 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:14:18.957913 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:14:18.957873 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:14:33.163359 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:14:33.163320 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:15:06.163539 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:15:06.163499 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:15:18.055136 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:15:18.055100 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:15:49.662956 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:15:49.662920 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:16:00.966177 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:16:00.966134 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:16:34.064423 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:16:34.064392 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:16:45.867429 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:16:45.867353 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:17:17.969074 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:17:17.969039 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:18:45.169687 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:18:45.169656 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:18:54.873131 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:18:54.873087 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:19:04.166555 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:19:04.166521 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:19:26.760639 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:19:26.760603 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:20:03.961440 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:20:03.961351 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:20:15.767567 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:20:15.767530 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:20:27.576175 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:20:27.576142 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:20:49.466609 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:20:49.466571 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:20:58.561510 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:20:58.561472 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:21:49.365607 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:21:49.365568 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:21:58.869267 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:21:58.869235 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:22:50.186892 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:22:50.186804 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:23:03.175513 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:03.175474 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:23:11.260494 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:11.260459 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:23:43.874103 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.874067 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p"] Jul 18 02:23:43.877586 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.877570 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:43.880034 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.880007 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f0bd15\"/\"openshift-service-ca.crt\"" Jul 18 02:23:43.880884 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.880864 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f0bd15\"/\"default-dockercfg-qg8lp\"" Jul 18 02:23:43.881006 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.880881 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f0bd15\"/\"test-model-f0bd15-kserve-self-signed-certs\"" Jul 18 02:23:43.881006 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.880903 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f0bd15\"/\"kube-root-ca.crt\"" Jul 18 02:23:43.887372 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.887348 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p"] Jul 18 02:23:43.967963 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.967928 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-home\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:43.967963 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.967963 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftd8z\" (UniqueName: \"kubernetes.io/projected/a3f28dff-2e14-4042-9790-fd7b5ec69268-kube-api-access-ftd8z\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:43.968211 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.967982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-dshm\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:43.968211 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.968038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-model-cache\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:43.968211 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.968080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f28dff-2e14-4042-9790-fd7b5ec69268-tls-certs\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:43.968211 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:43.968141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-tmp-dir\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.069198 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.069167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-home\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.069367 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.069205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftd8z\" (UniqueName: \"kubernetes.io/projected/a3f28dff-2e14-4042-9790-fd7b5ec69268-kube-api-access-ftd8z\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.069367 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.069221 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-dshm\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.069367 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.069355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-model-cache\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.069507 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.069390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f28dff-2e14-4042-9790-fd7b5ec69268-tls-certs\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.069507 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.069460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-tmp-dir\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.069806 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.069784 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-home\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.069932 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.069827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-model-cache\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.069932 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.069890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-tmp-dir\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.072351 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.072332 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-dshm\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.072647 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.072625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f28dff-2e14-4042-9790-fd7b5ec69268-tls-certs\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.077441 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.077416 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftd8z\" (UniqueName: \"kubernetes.io/projected/a3f28dff-2e14-4042-9790-fd7b5ec69268-kube-api-access-ftd8z\") pod \"test-model-f0bd15-kserve-7c75cdcfbf-7g89p\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.188916 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.188830 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:44.321229 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.321202 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p"] Jul 18 02:23:44.323864 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:23:44.323838 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3f28dff_2e14_4042_9790_fd7b5ec69268.slice/crio-397f92373dfd38b7a331d4efa525c12891a7ccf74a01d7e3d96d9e5acd4cc894 WatchSource:0}: Error finding container 397f92373dfd38b7a331d4efa525c12891a7ccf74a01d7e3d96d9e5acd4cc894: Status 404 returned error can't find the container with id 397f92373dfd38b7a331d4efa525c12891a7ccf74a01d7e3d96d9e5acd4cc894 Jul 18 02:23:44.738437 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.738401 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" event={"ID":"a3f28dff-2e14-4042-9790-fd7b5ec69268","Type":"ContainerStarted","Data":"578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83"} Jul 18 02:23:44.738605 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.738449 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" event={"ID":"a3f28dff-2e14-4042-9790-fd7b5ec69268","Type":"ContainerStarted","Data":"397f92373dfd38b7a331d4efa525c12891a7ccf74a01d7e3d96d9e5acd4cc894"} Jul 18 02:23:44.754712 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:44.754659 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" podStartSLOduration=1.7546434400000002 podStartE2EDuration="1.75464344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:23:44.753395893 +0000 UTC m=+2129.084636876" watchObservedRunningTime="2026-07-18 02:23:44.75464344 +0000 UTC m=+2129.085884421" Jul 18 02:23:48.263482 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:48.263443 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:23:54.196779 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:54.196729 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:54.196779 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:54.196791 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:54.203418 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:54.203400 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:23:54.796129 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:23:54.796100 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:24:06.489039 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.489001 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p"] Jul 18 02:24:06.622906 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.622827 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx"] Jul 18 02:24:06.626112 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.626094 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" Jul 18 02:24:06.628356 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.628335 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vpdlz\"" Jul 18 02:24:06.628483 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.628342 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 18 02:24:06.633477 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.633453 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx"] Jul 18 02:24:06.677032 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.676994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde72952-dd14-4b21-845d-88021c54972c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx\" (UID: \"cde72952-dd14-4b21-845d-88021c54972c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" Jul 18 02:24:06.778261 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.778168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde72952-dd14-4b21-845d-88021c54972c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx\" (UID: \"cde72952-dd14-4b21-845d-88021c54972c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" Jul 18 02:24:06.778829 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.778811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde72952-dd14-4b21-845d-88021c54972c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx\" (UID: \"cde72952-dd14-4b21-845d-88021c54972c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" Jul 18 02:24:06.937688 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:06.937651 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" Jul 18 02:24:07.066934 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:07.066864 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx"] Jul 18 02:24:07.069371 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:24:07.069348 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde72952_dd14_4b21_845d_88021c54972c.slice/crio-acac943644638b445acc61b4644dc1678eb7b3ec4c32ddc0017cb13b58adb5fe WatchSource:0}: Error finding container acac943644638b445acc61b4644dc1678eb7b3ec4c32ddc0017cb13b58adb5fe: Status 404 returned error can't find the container with id acac943644638b445acc61b4644dc1678eb7b3ec4c32ddc0017cb13b58adb5fe Jul 18 02:24:07.071786 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:07.071767 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 02:24:07.847040 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:07.847010 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" event={"ID":"cde72952-dd14-4b21-845d-88021c54972c","Type":"ContainerStarted","Data":"acac943644638b445acc61b4644dc1678eb7b3ec4c32ddc0017cb13b58adb5fe"} Jul 18 02:24:10.960578 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:10.960539 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:24:14.880678 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:14.880640 2582 generic.go:356] "Generic (PLEG): container finished" podID="cde72952-dd14-4b21-845d-88021c54972c" containerID="63f79ca53cc2ed53689b3e1f0e3298d3c39e7abff4c827ee040ec8ef93c5b13c" exitCode=0 Jul 18 02:24:14.881098 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:14.880718 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" event={"ID":"cde72952-dd14-4b21-845d-88021c54972c","Type":"ContainerDied","Data":"63f79ca53cc2ed53689b3e1f0e3298d3c39e7abff4c827ee040ec8ef93c5b13c"} Jul 18 02:24:16.013312 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:16.013289 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" Jul 18 02:24:16.062614 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:16.062580 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde72952-dd14-4b21-845d-88021c54972c-maas-api-service-ca\") pod \"cde72952-dd14-4b21-845d-88021c54972c\" (UID: \"cde72952-dd14-4b21-845d-88021c54972c\") " Jul 18 02:24:16.062981 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:16.062958 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde72952-dd14-4b21-845d-88021c54972c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cde72952-dd14-4b21-845d-88021c54972c" (UID: "cde72952-dd14-4b21-845d-88021c54972c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:24:16.163660 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:16.163580 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde72952-dd14-4b21-845d-88021c54972c-maas-api-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:24:16.891427 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:16.891387 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" event={"ID":"cde72952-dd14-4b21-845d-88021c54972c","Type":"ContainerDied","Data":"acac943644638b445acc61b4644dc1678eb7b3ec4c32ddc0017cb13b58adb5fe"} Jul 18 02:24:16.891427 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:16.891416 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx" Jul 18 02:24:16.891640 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:16.891483 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acac943644638b445acc61b4644dc1678eb7b3ec4c32ddc0017cb13b58adb5fe" Jul 18 02:24:21.512980 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.512932 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" podUID="a3f28dff-2e14-4042-9790-fd7b5ec69268" containerName="main" containerID="cri-o://578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83" gracePeriod=45 Jul 18 02:24:21.781455 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.781429 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:24:21.915493 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.915456 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftd8z\" (UniqueName: \"kubernetes.io/projected/a3f28dff-2e14-4042-9790-fd7b5ec69268-kube-api-access-ftd8z\") pod \"a3f28dff-2e14-4042-9790-fd7b5ec69268\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " Jul 18 02:24:21.915694 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.915530 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-home\") pod \"a3f28dff-2e14-4042-9790-fd7b5ec69268\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " Jul 18 02:24:21.915694 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.915622 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-tmp-dir\") pod \"a3f28dff-2e14-4042-9790-fd7b5ec69268\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " Jul 18 02:24:21.915694 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.915650 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f28dff-2e14-4042-9790-fd7b5ec69268-tls-certs\") pod \"a3f28dff-2e14-4042-9790-fd7b5ec69268\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " Jul 18 02:24:21.915694 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.915678 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-dshm\") pod \"a3f28dff-2e14-4042-9790-fd7b5ec69268\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " Jul 18 02:24:21.915927 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.915704 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-model-cache\") pod \"a3f28dff-2e14-4042-9790-fd7b5ec69268\" (UID: \"a3f28dff-2e14-4042-9790-fd7b5ec69268\") " Jul 18 02:24:21.916171 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.916095 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-home" (OuterVolumeSpecName: "home") pod "a3f28dff-2e14-4042-9790-fd7b5ec69268" (UID: "a3f28dff-2e14-4042-9790-fd7b5ec69268"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 02:24:21.916260 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.916196 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3f28dff-2e14-4042-9790-fd7b5ec69268" (UID: "a3f28dff-2e14-4042-9790-fd7b5ec69268"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 02:24:21.916389 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.916354 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3f28dff-2e14-4042-9790-fd7b5ec69268" (UID: "a3f28dff-2e14-4042-9790-fd7b5ec69268"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 02:24:21.919704 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.919675 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f28dff-2e14-4042-9790-fd7b5ec69268-kube-api-access-ftd8z" (OuterVolumeSpecName: "kube-api-access-ftd8z") pod "a3f28dff-2e14-4042-9790-fd7b5ec69268" (UID: "a3f28dff-2e14-4042-9790-fd7b5ec69268"). InnerVolumeSpecName "kube-api-access-ftd8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:24:21.919942 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.919904 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f28dff-2e14-4042-9790-fd7b5ec69268-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3f28dff-2e14-4042-9790-fd7b5ec69268" (UID: "a3f28dff-2e14-4042-9790-fd7b5ec69268"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:24:21.920122 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.920090 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-dshm" (OuterVolumeSpecName: "dshm") pod "a3f28dff-2e14-4042-9790-fd7b5ec69268" (UID: "a3f28dff-2e14-4042-9790-fd7b5ec69268"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 02:24:21.920201 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.920101 2582 generic.go:356] "Generic (PLEG): container finished" podID="a3f28dff-2e14-4042-9790-fd7b5ec69268" containerID="578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83" exitCode=0 Jul 18 02:24:21.920201 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.920127 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" event={"ID":"a3f28dff-2e14-4042-9790-fd7b5ec69268","Type":"ContainerDied","Data":"578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83"} Jul 18 02:24:21.920201 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.920158 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" Jul 18 02:24:21.920201 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.920168 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p" event={"ID":"a3f28dff-2e14-4042-9790-fd7b5ec69268","Type":"ContainerDied","Data":"397f92373dfd38b7a331d4efa525c12891a7ccf74a01d7e3d96d9e5acd4cc894"} Jul 18 02:24:21.920366 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.920186 2582 scope.go:117] "RemoveContainer" containerID="578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83" Jul 18 02:24:21.936228 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.936211 2582 scope.go:117] "RemoveContainer" containerID="578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83" Jul 18 02:24:21.936507 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:24:21.936489 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83\": container with ID starting with 578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83 not found: ID does not exist" containerID="578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83" Jul 18 02:24:21.936573 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.936519 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83"} err="failed to get container status \"578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83\": rpc error: code = NotFound desc = could not find container \"578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83\": container with ID starting with 578da87402c7ab31b5f0da4ef26b306bd6729055b941ad96cce1a4cc0ca0ab83 not found: ID does not exist" Jul 18 02:24:21.947111 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.947045 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p"] Jul 18 02:24:21.948876 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:21.948841 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f0bd15/test-model-f0bd15-kserve-7c75cdcfbf-7g89p"] Jul 18 02:24:22.016651 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:22.016569 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-home\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:24:22.016651 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:22.016597 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-tmp-dir\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:24:22.016651 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:22.016605 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f28dff-2e14-4042-9790-fd7b5ec69268-tls-certs\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:24:22.016651 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:22.016614 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-dshm\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:24:22.016651 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:22.016623 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f28dff-2e14-4042-9790-fd7b5ec69268-model-cache\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:24:22.016651 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:22.016632 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftd8z\" (UniqueName: \"kubernetes.io/projected/a3f28dff-2e14-4042-9790-fd7b5ec69268-kube-api-access-ftd8z\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:24:22.195046 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:22.195014 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f28dff-2e14-4042-9790-fd7b5ec69268" path="/var/lib/kubelet/pods/a3f28dff-2e14-4042-9790-fd7b5ec69268/volumes" Jul 18 02:24:29.967465 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:29.967432 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:24:31.713089 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:31.713048 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx"] Jul 18 02:24:31.715902 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:31.715873 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c89a1-bpwtx"] Jul 18 02:24:32.195479 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:32.195444 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde72952-dd14-4b21-845d-88021c54972c" path="/var/lib/kubelet/pods/cde72952-dd14-4b21-845d-88021c54972c/volumes" Jul 18 02:24:38.473541 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.473502 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p"] Jul 18 02:24:38.474105 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.474083 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde72952-dd14-4b21-845d-88021c54972c" containerName="revoke-keys" Jul 18 02:24:38.474186 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.474108 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde72952-dd14-4b21-845d-88021c54972c" containerName="revoke-keys" Jul 18 02:24:38.474186 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.474146 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f28dff-2e14-4042-9790-fd7b5ec69268" containerName="main" Jul 18 02:24:38.474186 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.474156 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f28dff-2e14-4042-9790-fd7b5ec69268" containerName="main" Jul 18 02:24:38.474327 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.474236 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f28dff-2e14-4042-9790-fd7b5ec69268" containerName="main" Jul 18 02:24:38.474327 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.474254 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde72952-dd14-4b21-845d-88021c54972c" containerName="revoke-keys" Jul 18 02:24:38.478568 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.478547 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" Jul 18 02:24:38.481007 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.480985 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vpdlz\"" Jul 18 02:24:38.481105 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.480989 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 18 02:24:38.484445 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.484425 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p"] Jul 18 02:24:38.567543 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.567510 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eb3da5-c202-44dc-a08b-c335acd6e6e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p\" (UID: \"43eb3da5-c202-44dc-a08b-c335acd6e6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" Jul 18 02:24:38.668992 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.668962 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eb3da5-c202-44dc-a08b-c335acd6e6e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p\" (UID: \"43eb3da5-c202-44dc-a08b-c335acd6e6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" Jul 18 02:24:38.669579 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.669558 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eb3da5-c202-44dc-a08b-c335acd6e6e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p\" (UID: \"43eb3da5-c202-44dc-a08b-c335acd6e6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" Jul 18 02:24:38.789776 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.789658 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" Jul 18 02:24:38.920573 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:38.920545 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p"] Jul 18 02:24:38.922753 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:24:38.922710 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43eb3da5_c202_44dc_a08b_c335acd6e6e0.slice/crio-1e3501630826d4d9a15f946098f98a74b1506b34454cd9ca64c63c801b96e9c0 WatchSource:0}: Error finding container 1e3501630826d4d9a15f946098f98a74b1506b34454cd9ca64c63c801b96e9c0: Status 404 returned error can't find the container with id 1e3501630826d4d9a15f946098f98a74b1506b34454cd9ca64c63c801b96e9c0 Jul 18 02:24:39.000376 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:39.000334 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" event={"ID":"43eb3da5-c202-44dc-a08b-c335acd6e6e0","Type":"ContainerStarted","Data":"1e3501630826d4d9a15f946098f98a74b1506b34454cd9ca64c63c801b96e9c0"} Jul 18 02:24:40.006595 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:40.006559 2582 generic.go:356] "Generic (PLEG): container finished" podID="43eb3da5-c202-44dc-a08b-c335acd6e6e0" containerID="21c27972089efe6a86ad715185a42c93bdfe04eccc2ca9b5bcab56f90df68781" exitCode=0 Jul 18 02:24:40.007050 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:40.006613 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" event={"ID":"43eb3da5-c202-44dc-a08b-c335acd6e6e0","Type":"ContainerDied","Data":"21c27972089efe6a86ad715185a42c93bdfe04eccc2ca9b5bcab56f90df68781"} Jul 18 02:24:41.139201 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:41.139175 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" Jul 18 02:24:41.290153 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:41.290065 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eb3da5-c202-44dc-a08b-c335acd6e6e0-maas-api-service-ca\") pod \"43eb3da5-c202-44dc-a08b-c335acd6e6e0\" (UID: \"43eb3da5-c202-44dc-a08b-c335acd6e6e0\") " Jul 18 02:24:41.290439 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:41.290412 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43eb3da5-c202-44dc-a08b-c335acd6e6e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43eb3da5-c202-44dc-a08b-c335acd6e6e0" (UID: "43eb3da5-c202-44dc-a08b-c335acd6e6e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:24:41.391283 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:41.391245 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eb3da5-c202-44dc-a08b-c335acd6e6e0-maas-api-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:24:42.017248 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:42.017213 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" Jul 18 02:24:42.017418 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:42.017222 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p" event={"ID":"43eb3da5-c202-44dc-a08b-c335acd6e6e0","Type":"ContainerDied","Data":"1e3501630826d4d9a15f946098f98a74b1506b34454cd9ca64c63c801b96e9c0"} Jul 18 02:24:42.017418 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:42.017370 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e3501630826d4d9a15f946098f98a74b1506b34454cd9ca64c63c801b96e9c0" Jul 18 02:24:51.965460 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:51.965430 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:24:53.559180 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:53.559136 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p"] Jul 18 02:24:53.565263 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:53.565232 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bd15-29s8p"] Jul 18 02:24:54.194656 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:24:54.194623 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43eb3da5-c202-44dc-a08b-c335acd6e6e0" path="/var/lib/kubelet/pods/43eb3da5-c202-44dc-a08b-c335acd6e6e0/volumes" Jul 18 02:25:11.671516 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:11.671479 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:25:12.164289 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.164257 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w"] Jul 18 02:25:12.164836 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.164813 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43eb3da5-c202-44dc-a08b-c335acd6e6e0" containerName="revoke-keys" Jul 18 02:25:12.164836 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.164832 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="43eb3da5-c202-44dc-a08b-c335acd6e6e0" containerName="revoke-keys" Jul 18 02:25:12.164986 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.164912 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="43eb3da5-c202-44dc-a08b-c335acd6e6e0" containerName="revoke-keys" Jul 18 02:25:12.170364 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.170337 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.172962 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.172933 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 18 02:25:12.172962 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.172935 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 18 02:25:12.172962 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.172931 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a724ed-serving-cert\"" Jul 18 02:25:12.173703 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.173686 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4q4w8\"" Jul 18 02:25:12.179199 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.178770 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w"] Jul 18 02:25:12.260071 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.260027 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ff66ae-98af-4733-94c3-4d9fa423f0eb-maas-api-tls\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.260284 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.260246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbfs\" (UniqueName: \"kubernetes.io/projected/61ff66ae-98af-4733-94c3-4d9fa423f0eb-kube-api-access-jvbfs\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.260382 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.260285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ff66ae-98af-4733-94c3-4d9fa423f0eb-openshift-service-ca\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.361183 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.361141 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ff66ae-98af-4733-94c3-4d9fa423f0eb-maas-api-tls\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.361383 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.361248 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbfs\" (UniqueName: \"kubernetes.io/projected/61ff66ae-98af-4733-94c3-4d9fa423f0eb-kube-api-access-jvbfs\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.361383 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.361278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ff66ae-98af-4733-94c3-4d9fa423f0eb-openshift-service-ca\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.361383 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:25:12.361312 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-serving-cert: secret "maas-api-e2e-api-b-a724ed-serving-cert" not found Jul 18 02:25:12.361550 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:25:12.361397 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ff66ae-98af-4733-94c3-4d9fa423f0eb-maas-api-tls podName:61ff66ae-98af-4733-94c3-4d9fa423f0eb nodeName:}" failed. No retries permitted until 2026-07-18 02:25:12.861373104 +0000 UTC m=+2217.192614064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61ff66ae-98af-4733-94c3-4d9fa423f0eb-maas-api-tls") pod "maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" (UID: "61ff66ae-98af-4733-94c3-4d9fa423f0eb") : secret "maas-api-e2e-api-b-a724ed-serving-cert" not found Jul 18 02:25:12.361967 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.361945 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ff66ae-98af-4733-94c3-4d9fa423f0eb-openshift-service-ca\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.381649 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.381624 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbfs\" (UniqueName: \"kubernetes.io/projected/61ff66ae-98af-4733-94c3-4d9fa423f0eb-kube-api-access-jvbfs\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.866754 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.866716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ff66ae-98af-4733-94c3-4d9fa423f0eb-maas-api-tls\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:12.869250 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:12.869228 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ff66ae-98af-4733-94c3-4d9fa423f0eb-maas-api-tls\") pod \"maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:13.083310 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:13.083273 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:13.215625 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:13.215601 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w"] Jul 18 02:25:13.217452 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:25:13.217423 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ff66ae_98af_4733_94c3_4d9fa423f0eb.slice/crio-09a906dcbb1f734d208845a604dff2cf929db100236ea09d37cb808e3490cdb9 WatchSource:0}: Error finding container 09a906dcbb1f734d208845a604dff2cf929db100236ea09d37cb808e3490cdb9: Status 404 returned error can't find the container with id 09a906dcbb1f734d208845a604dff2cf929db100236ea09d37cb808e3490cdb9 Jul 18 02:25:14.177332 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:14.177293 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" event={"ID":"61ff66ae-98af-4733-94c3-4d9fa423f0eb","Type":"ContainerStarted","Data":"09a906dcbb1f734d208845a604dff2cf929db100236ea09d37cb808e3490cdb9"} Jul 18 02:25:16.194752 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:16.194708 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" event={"ID":"61ff66ae-98af-4733-94c3-4d9fa423f0eb","Type":"ContainerStarted","Data":"273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea"} Jul 18 02:25:16.195151 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:16.194789 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:16.208805 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:16.208716 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" podStartSLOduration=2.15825829 podStartE2EDuration="4.208700688s" totalImagesPullingTime="2.050442398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:25:16.207093391 +0000 UTC m=+2220.538334623" watchObservedRunningTime="2026-07-18 02:25:16.208700688 +0000 UTC m=+2220.539941670" Jul 18 02:25:22.199569 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:22.199540 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:25:24.408328 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:24.408293 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8"] Jul 18 02:25:24.411547 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:24.411531 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" Jul 18 02:25:24.413967 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:24.413950 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vpdlz\"" Jul 18 02:25:24.419950 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:24.419926 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8"] Jul 18 02:25:24.572401 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:24.572358 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648fb9e4-b0c2-4fa4-8bf4-c01051a17791-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8\" (UID: \"648fb9e4-b0c2-4fa4-8bf4-c01051a17791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" Jul 18 02:25:24.673822 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:24.673701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648fb9e4-b0c2-4fa4-8bf4-c01051a17791-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8\" (UID: \"648fb9e4-b0c2-4fa4-8bf4-c01051a17791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" Jul 18 02:25:24.674331 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:24.674310 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648fb9e4-b0c2-4fa4-8bf4-c01051a17791-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8\" (UID: \"648fb9e4-b0c2-4fa4-8bf4-c01051a17791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" Jul 18 02:25:24.722414 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:24.722373 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" Jul 18 02:25:24.857436 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:24.857412 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8"] Jul 18 02:25:24.859416 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:25:24.859390 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648fb9e4_b0c2_4fa4_8bf4_c01051a17791.slice/crio-831720f8cc678a7330bae6ba25add9ddc113311e4be620d36872437fefc33e54 WatchSource:0}: Error finding container 831720f8cc678a7330bae6ba25add9ddc113311e4be620d36872437fefc33e54: Status 404 returned error can't find the container with id 831720f8cc678a7330bae6ba25add9ddc113311e4be620d36872437fefc33e54 Jul 18 02:25:25.232869 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:25.232830 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" event={"ID":"648fb9e4-b0c2-4fa4-8bf4-c01051a17791","Type":"ContainerStarted","Data":"8de8ff7c1a152669859e8719d9e8eae22c75c4fb0a7e45bd11921e0b8974c07d"} Jul 18 02:25:25.233050 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:25.232888 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" event={"ID":"648fb9e4-b0c2-4fa4-8bf4-c01051a17791","Type":"ContainerStarted","Data":"831720f8cc678a7330bae6ba25add9ddc113311e4be620d36872437fefc33e54"} Jul 18 02:25:25.248552 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:25.248507 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" podStartSLOduration=1.248490367 podStartE2EDuration="1.248490367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:25:25.247572099 +0000 UTC m=+2229.578813084" watchObservedRunningTime="2026-07-18 02:25:25.248490367 +0000 UTC m=+2229.579731348" Jul 18 02:25:26.239553 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:26.239523 2582 generic.go:356] "Generic (PLEG): container finished" podID="648fb9e4-b0c2-4fa4-8bf4-c01051a17791" containerID="8de8ff7c1a152669859e8719d9e8eae22c75c4fb0a7e45bd11921e0b8974c07d" exitCode=0 Jul 18 02:25:26.239972 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:26.239612 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" event={"ID":"648fb9e4-b0c2-4fa4-8bf4-c01051a17791","Type":"ContainerDied","Data":"8de8ff7c1a152669859e8719d9e8eae22c75c4fb0a7e45bd11921e0b8974c07d"} Jul 18 02:25:27.272136 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:27.272102 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:25:27.376818 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:27.376794 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" Jul 18 02:25:27.502331 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:27.502242 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648fb9e4-b0c2-4fa4-8bf4-c01051a17791-maas-api-service-ca\") pod \"648fb9e4-b0c2-4fa4-8bf4-c01051a17791\" (UID: \"648fb9e4-b0c2-4fa4-8bf4-c01051a17791\") " Jul 18 02:25:27.502574 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:27.502552 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648fb9e4-b0c2-4fa4-8bf4-c01051a17791-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "648fb9e4-b0c2-4fa4-8bf4-c01051a17791" (UID: "648fb9e4-b0c2-4fa4-8bf4-c01051a17791"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:25:27.603474 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:27.603448 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648fb9e4-b0c2-4fa4-8bf4-c01051a17791-maas-api-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:25:28.250050 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:28.250018 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" event={"ID":"648fb9e4-b0c2-4fa4-8bf4-c01051a17791","Type":"ContainerDied","Data":"831720f8cc678a7330bae6ba25add9ddc113311e4be620d36872437fefc33e54"} Jul 18 02:25:28.250050 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:28.250042 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8" Jul 18 02:25:28.250252 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:28.250066 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="831720f8cc678a7330bae6ba25add9ddc113311e4be620d36872437fefc33e54" Jul 18 02:25:42.965349 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:42.965263 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:25:44.562845 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:44.562808 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8"] Jul 18 02:25:44.564293 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:44.564271 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a3654-w8qr8"] Jul 18 02:25:46.194327 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:46.194292 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="648fb9e4-b0c2-4fa4-8bf4-c01051a17791" path="/var/lib/kubelet/pods/648fb9e4-b0c2-4fa4-8bf4-c01051a17791/volumes" Jul 18 02:25:51.218993 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.218959 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx"] Jul 18 02:25:51.219353 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.219316 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648fb9e4-b0c2-4fa4-8bf4-c01051a17791" containerName="revoke-keys" Jul 18 02:25:51.219353 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.219326 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="648fb9e4-b0c2-4fa4-8bf4-c01051a17791" containerName="revoke-keys" Jul 18 02:25:51.219436 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.219378 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="648fb9e4-b0c2-4fa4-8bf4-c01051a17791" containerName="revoke-keys" Jul 18 02:25:51.222173 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.222156 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" Jul 18 02:25:51.225990 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.225961 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vpdlz\"" Jul 18 02:25:51.235344 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.235324 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx"] Jul 18 02:25:51.312048 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.312009 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c50c679-f5c9-4526-b573-ba61f81b33e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx\" (UID: \"0c50c679-f5c9-4526-b573-ba61f81b33e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" Jul 18 02:25:51.413380 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.413344 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c50c679-f5c9-4526-b573-ba61f81b33e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx\" (UID: \"0c50c679-f5c9-4526-b573-ba61f81b33e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" Jul 18 02:25:51.414003 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.413982 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c50c679-f5c9-4526-b573-ba61f81b33e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx\" (UID: \"0c50c679-f5c9-4526-b573-ba61f81b33e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" Jul 18 02:25:51.531675 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.531587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" Jul 18 02:25:51.665782 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:51.665723 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx"] Jul 18 02:25:51.668821 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:25:51.668794 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c50c679_f5c9_4526_b573_ba61f81b33e1.slice/crio-6ef7c133843b407bd7cc9c65389ef2784195e851f79c104c217de861d51358a5 WatchSource:0}: Error finding container 6ef7c133843b407bd7cc9c65389ef2784195e851f79c104c217de861d51358a5: Status 404 returned error can't find the container with id 6ef7c133843b407bd7cc9c65389ef2784195e851f79c104c217de861d51358a5 Jul 18 02:25:52.367981 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:52.367948 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" event={"ID":"0c50c679-f5c9-4526-b573-ba61f81b33e1","Type":"ContainerStarted","Data":"0a3977fc20789de5ca1c5af9b71f386106fc61b42fb23342cce17fdd5f55fb4c"} Jul 18 02:25:52.368370 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:52.367998 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" event={"ID":"0c50c679-f5c9-4526-b573-ba61f81b33e1","Type":"ContainerStarted","Data":"6ef7c133843b407bd7cc9c65389ef2784195e851f79c104c217de861d51358a5"} Jul 18 02:25:52.384976 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:52.384923 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" podStartSLOduration=1.384904069 podStartE2EDuration="1.384904069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:25:52.384399796 +0000 UTC m=+2256.715640777" watchObservedRunningTime="2026-07-18 02:25:52.384904069 +0000 UTC m=+2256.716145053" Jul 18 02:25:53.374127 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:53.374089 2582 generic.go:356] "Generic (PLEG): container finished" podID="0c50c679-f5c9-4526-b573-ba61f81b33e1" containerID="0a3977fc20789de5ca1c5af9b71f386106fc61b42fb23342cce17fdd5f55fb4c" exitCode=0 Jul 18 02:25:53.374515 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:53.374171 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" event={"ID":"0c50c679-f5c9-4526-b573-ba61f81b33e1","Type":"ContainerDied","Data":"0a3977fc20789de5ca1c5af9b71f386106fc61b42fb23342cce17fdd5f55fb4c"} Jul 18 02:25:54.512497 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:54.512472 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" Jul 18 02:25:54.644152 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:54.644061 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c50c679-f5c9-4526-b573-ba61f81b33e1-maas-api-service-ca\") pod \"0c50c679-f5c9-4526-b573-ba61f81b33e1\" (UID: \"0c50c679-f5c9-4526-b573-ba61f81b33e1\") " Jul 18 02:25:54.644427 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:54.644404 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c50c679-f5c9-4526-b573-ba61f81b33e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c50c679-f5c9-4526-b573-ba61f81b33e1" (UID: "0c50c679-f5c9-4526-b573-ba61f81b33e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:25:54.745145 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:54.745114 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c50c679-f5c9-4526-b573-ba61f81b33e1-maas-api-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:25:55.388883 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:55.388856 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" Jul 18 02:25:55.389118 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:55.388881 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx" event={"ID":"0c50c679-f5c9-4526-b573-ba61f81b33e1","Type":"ContainerDied","Data":"6ef7c133843b407bd7cc9c65389ef2784195e851f79c104c217de861d51358a5"} Jul 18 02:25:55.389118 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:25:55.388928 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ef7c133843b407bd7cc9c65389ef2784195e851f79c104c217de861d51358a5" Jul 18 02:26:06.650281 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:06.650246 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w"] Jul 18 02:26:06.650794 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:06.650475 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" podUID="61ff66ae-98af-4733-94c3-4d9fa423f0eb" containerName="maas-api" containerID="cri-o://273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea" gracePeriod=30 Jul 18 02:26:06.906612 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:06.906549 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:26:07.055599 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.055564 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvbfs\" (UniqueName: \"kubernetes.io/projected/61ff66ae-98af-4733-94c3-4d9fa423f0eb-kube-api-access-jvbfs\") pod \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " Jul 18 02:26:07.055816 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.055675 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ff66ae-98af-4733-94c3-4d9fa423f0eb-openshift-service-ca\") pod \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " Jul 18 02:26:07.055816 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.055796 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ff66ae-98af-4733-94c3-4d9fa423f0eb-maas-api-tls\") pod \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\" (UID: \"61ff66ae-98af-4733-94c3-4d9fa423f0eb\") " Jul 18 02:26:07.056105 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.056082 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ff66ae-98af-4733-94c3-4d9fa423f0eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61ff66ae-98af-4733-94c3-4d9fa423f0eb" (UID: "61ff66ae-98af-4733-94c3-4d9fa423f0eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:26:07.057961 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.057933 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ff66ae-98af-4733-94c3-4d9fa423f0eb-kube-api-access-jvbfs" (OuterVolumeSpecName: "kube-api-access-jvbfs") pod "61ff66ae-98af-4733-94c3-4d9fa423f0eb" (UID: "61ff66ae-98af-4733-94c3-4d9fa423f0eb"). InnerVolumeSpecName "kube-api-access-jvbfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:26:07.058062 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.057964 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ff66ae-98af-4733-94c3-4d9fa423f0eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61ff66ae-98af-4733-94c3-4d9fa423f0eb" (UID: "61ff66ae-98af-4733-94c3-4d9fa423f0eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:26:07.157448 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.157357 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ff66ae-98af-4733-94c3-4d9fa423f0eb-maas-api-tls\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:26:07.157448 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.157391 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvbfs\" (UniqueName: \"kubernetes.io/projected/61ff66ae-98af-4733-94c3-4d9fa423f0eb-kube-api-access-jvbfs\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:26:07.157448 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.157407 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ff66ae-98af-4733-94c3-4d9fa423f0eb-openshift-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:26:07.446646 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.446551 2582 generic.go:356] "Generic (PLEG): container finished" podID="61ff66ae-98af-4733-94c3-4d9fa423f0eb" containerID="273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea" exitCode=0 Jul 18 02:26:07.446646 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.446607 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" Jul 18 02:26:07.446861 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.446641 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" event={"ID":"61ff66ae-98af-4733-94c3-4d9fa423f0eb","Type":"ContainerDied","Data":"273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea"} Jul 18 02:26:07.446861 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.446701 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w" event={"ID":"61ff66ae-98af-4733-94c3-4d9fa423f0eb","Type":"ContainerDied","Data":"09a906dcbb1f734d208845a604dff2cf929db100236ea09d37cb808e3490cdb9"} Jul 18 02:26:07.446861 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.446731 2582 scope.go:117] "RemoveContainer" containerID="273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea" Jul 18 02:26:07.458811 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.458787 2582 scope.go:117] "RemoveContainer" containerID="273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea" Jul 18 02:26:07.459135 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:26:07.459114 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea\": container with ID starting with 273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea not found: ID does not exist" containerID="273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea" Jul 18 02:26:07.459196 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.459144 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea"} err="failed to get container status \"273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea\": rpc error: code = NotFound desc = could not find container \"273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea\": container with ID starting with 273c7629d17ae9094307f0adcac3c0cb649df143e8c97d9497d6e185b75d18ea not found: ID does not exist" Jul 18 02:26:07.468217 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.468190 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w"] Jul 18 02:26:07.471695 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:07.471672 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a724ed-6bddcdb4d6-crp7w"] Jul 18 02:26:08.195806 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:08.195773 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ff66ae-98af-4733-94c3-4d9fa423f0eb" path="/var/lib/kubelet/pods/61ff66ae-98af-4733-94c3-4d9fa423f0eb/volumes" Jul 18 02:26:11.293209 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:11.293176 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx"] Jul 18 02:26:11.297752 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:11.297711 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a724ed-tjnzx"] Jul 18 02:26:11.505634 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:11.505600 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:26:12.196137 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:12.196097 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c50c679-f5c9-4526-b573-ba61f81b33e1" path="/var/lib/kubelet/pods/0c50c679-f5c9-4526-b573-ba61f81b33e1/volumes" Jul 18 02:26:22.507277 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.507233 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7"] Jul 18 02:26:22.507676 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.507570 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c50c679-f5c9-4526-b573-ba61f81b33e1" containerName="revoke-keys" Jul 18 02:26:22.507676 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.507580 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c50c679-f5c9-4526-b573-ba61f81b33e1" containerName="revoke-keys" Jul 18 02:26:22.507676 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.507595 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ff66ae-98af-4733-94c3-4d9fa423f0eb" containerName="maas-api" Jul 18 02:26:22.507676 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.507600 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ff66ae-98af-4733-94c3-4d9fa423f0eb" containerName="maas-api" Jul 18 02:26:22.507676 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.507661 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ff66ae-98af-4733-94c3-4d9fa423f0eb" containerName="maas-api" Jul 18 02:26:22.507676 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.507670 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c50c679-f5c9-4526-b573-ba61f81b33e1" containerName="revoke-keys" Jul 18 02:26:22.511030 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.511010 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:22.513715 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.513684 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 18 02:26:22.513898 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.513866 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4q4w8\"" Jul 18 02:26:22.514659 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.514645 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-deec4e-serving-cert\"" Jul 18 02:26:22.514728 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.514673 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 18 02:26:22.522444 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.522422 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7"] Jul 18 02:26:22.600842 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.600805 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbf07af-1216-4c15-8434-be033e89cd27-openshift-service-ca\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:22.601021 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.600874 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:22.601021 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.600936 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv8br\" (UniqueName: \"kubernetes.io/projected/fbbf07af-1216-4c15-8434-be033e89cd27-kube-api-access-xv8br\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:22.702158 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.702116 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv8br\" (UniqueName: \"kubernetes.io/projected/fbbf07af-1216-4c15-8434-be033e89cd27-kube-api-access-xv8br\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:22.702329 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.702231 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbf07af-1216-4c15-8434-be033e89cd27-openshift-service-ca\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:22.702329 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.702280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:22.702409 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:26:22.702395 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-serving-cert: secret "maas-api-e2e-shared-a-deec4e-serving-cert" not found Jul 18 02:26:22.702482 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:26:22.702472 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls podName:fbbf07af-1216-4c15-8434-be033e89cd27 nodeName:}" failed. No retries permitted until 2026-07-18 02:26:23.202452158 +0000 UTC m=+2287.533693129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls") pod "maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" (UID: "fbbf07af-1216-4c15-8434-be033e89cd27") : secret "maas-api-e2e-shared-a-deec4e-serving-cert" not found Jul 18 02:26:22.703160 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.703141 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbf07af-1216-4c15-8434-be033e89cd27-openshift-service-ca\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:22.711441 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:22.711419 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv8br\" (UniqueName: \"kubernetes.io/projected/fbbf07af-1216-4c15-8434-be033e89cd27-kube-api-access-xv8br\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:23.206954 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:23.206918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:23.207133 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:26:23.207037 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-serving-cert: secret "maas-api-e2e-shared-a-deec4e-serving-cert" not found Jul 18 02:26:23.207133 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:26:23.207089 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls podName:fbbf07af-1216-4c15-8434-be033e89cd27 nodeName:}" failed. No retries permitted until 2026-07-18 02:26:24.207075367 +0000 UTC m=+2288.538316327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls") pod "maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" (UID: "fbbf07af-1216-4c15-8434-be033e89cd27") : secret "maas-api-e2e-shared-a-deec4e-serving-cert" not found Jul 18 02:26:24.217546 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:24.217512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:24.220127 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:24.220094 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls\") pod \"maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:24.321769 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:24.321716 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:24.454939 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:24.454914 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7"] Jul 18 02:26:24.457104 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:26:24.457076 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbf07af_1216_4c15_8434_be033e89cd27.slice/crio-3f48b13ec416f409c4a7e707feac45323898e34d37048792a18e571a96f250c2 WatchSource:0}: Error finding container 3f48b13ec416f409c4a7e707feac45323898e34d37048792a18e571a96f250c2: Status 404 returned error can't find the container with id 3f48b13ec416f409c4a7e707feac45323898e34d37048792a18e571a96f250c2 Jul 18 02:26:24.528461 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:24.528381 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" event={"ID":"fbbf07af-1216-4c15-8434-be033e89cd27","Type":"ContainerStarted","Data":"3f48b13ec416f409c4a7e707feac45323898e34d37048792a18e571a96f250c2"} Jul 18 02:26:25.534837 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:25.534793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" event={"ID":"fbbf07af-1216-4c15-8434-be033e89cd27","Type":"ContainerStarted","Data":"902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70"} Jul 18 02:26:25.535213 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:25.534870 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:25.553065 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:25.553010 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" podStartSLOduration=3.152140037 podStartE2EDuration="3.552996492s" totalImagesPullingTime="400.856455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:26:25.551017946 +0000 UTC m=+2289.882258991" watchObservedRunningTime="2026-07-18 02:26:25.552996492 +0000 UTC m=+2289.884237473" Jul 18 02:26:30.429946 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.429905 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4"] Jul 18 02:26:30.434538 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.434514 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:26:30.434717 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.434626 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:30.438634 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.438613 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-2a8451-serving-cert\"" Jul 18 02:26:30.457171 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.457137 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4"] Jul 18 02:26:30.576024 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.575989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3fbdd1-de53-4379-8618-35e256f1e536-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:30.576195 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.576178 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3fbdd1-de53-4379-8618-35e256f1e536-maas-api-tls\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:30.576238 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.576223 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79sw6\" (UniqueName: \"kubernetes.io/projected/ee3fbdd1-de53-4379-8618-35e256f1e536-kube-api-access-79sw6\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:30.677445 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.677410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3fbdd1-de53-4379-8618-35e256f1e536-maas-api-tls\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:30.677628 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.677453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79sw6\" (UniqueName: \"kubernetes.io/projected/ee3fbdd1-de53-4379-8618-35e256f1e536-kube-api-access-79sw6\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:30.677628 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.677482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3fbdd1-de53-4379-8618-35e256f1e536-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:30.677628 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:26:30.677577 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-serving-cert: secret "maas-api-e2e-shared-b-2a8451-serving-cert" not found Jul 18 02:26:30.677870 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:26:30.677645 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee3fbdd1-de53-4379-8618-35e256f1e536-maas-api-tls podName:ee3fbdd1-de53-4379-8618-35e256f1e536 nodeName:}" failed. No retries permitted until 2026-07-18 02:26:31.177625955 +0000 UTC m=+2295.508866917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee3fbdd1-de53-4379-8618-35e256f1e536-maas-api-tls") pod "maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" (UID: "ee3fbdd1-de53-4379-8618-35e256f1e536") : secret "maas-api-e2e-shared-b-2a8451-serving-cert" not found Jul 18 02:26:30.678094 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.678077 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3fbdd1-de53-4379-8618-35e256f1e536-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:30.688425 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:30.688371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79sw6\" (UniqueName: \"kubernetes.io/projected/ee3fbdd1-de53-4379-8618-35e256f1e536-kube-api-access-79sw6\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:31.181662 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:31.181620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3fbdd1-de53-4379-8618-35e256f1e536-maas-api-tls\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:31.184388 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:31.184363 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3fbdd1-de53-4379-8618-35e256f1e536-maas-api-tls\") pod \"maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:31.345691 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:31.345656 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:31.479271 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:31.479246 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4"] Jul 18 02:26:31.481546 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:26:31.481514 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3fbdd1_de53_4379_8618_35e256f1e536.slice/crio-debe45db22faab68834a609c0bb719ce5f5b298ab11108e28f8849ae5656123f WatchSource:0}: Error finding container debe45db22faab68834a609c0bb719ce5f5b298ab11108e28f8849ae5656123f: Status 404 returned error can't find the container with id debe45db22faab68834a609c0bb719ce5f5b298ab11108e28f8849ae5656123f Jul 18 02:26:31.543795 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:31.543768 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:26:31.565075 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:31.565043 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" event={"ID":"ee3fbdd1-de53-4379-8618-35e256f1e536","Type":"ContainerStarted","Data":"debe45db22faab68834a609c0bb719ce5f5b298ab11108e28f8849ae5656123f"} Jul 18 02:26:32.571598 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:32.571567 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" event={"ID":"ee3fbdd1-de53-4379-8618-35e256f1e536","Type":"ContainerStarted","Data":"a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7"} Jul 18 02:26:32.572008 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:32.571665 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:32.587169 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:32.587117 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" podStartSLOduration=2.3168703600000002 podStartE2EDuration="2.587098814s" totalImagesPullingTime="270.228454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:26:32.586185482 +0000 UTC m=+2296.917426461" watchObservedRunningTime="2026-07-18 02:26:32.587098814 +0000 UTC m=+2296.918339797" Jul 18 02:26:38.582519 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:38.582491 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:26:41.675885 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:41.675851 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:26:49.565459 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:26:49.565414 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:27:06.969686 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:27:06.969595 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:27:21.763401 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:27:21.763366 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:27:39.963685 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:27:39.963650 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:27:54.461238 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:27:54.461201 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:28:10.662443 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:28:10.662407 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:28:28.072881 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:28:28.072837 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:28:47.976376 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:28:47.976292 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:29:06.863964 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:29:06.863932 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:29:21.164416 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:29:21.164382 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:29:35.669209 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:29:35.669168 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:30:00.151665 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.151628 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5"] Jul 18 02:30:00.155598 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.155577 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg"] Jul 18 02:30:00.155718 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.155694 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5" Jul 18 02:30:00.158469 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.158448 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vpdlz\"" Jul 18 02:30:00.158766 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.158751 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2"] Jul 18 02:30:00.158840 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.158813 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg" Jul 18 02:30:00.162029 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.162013 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2" Jul 18 02:30:00.167240 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.167223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5" Jul 18 02:30:00.169354 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.169336 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5"] Jul 18 02:30:00.172699 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.171681 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2"] Jul 18 02:30:00.174006 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.173982 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg"] Jul 18 02:30:00.178305 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.178285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg" Jul 18 02:30:00.215815 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.215790 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2" Jul 18 02:30:00.337331 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.337157 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5"] Jul 18 02:30:00.340798 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:30:00.340726 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb919842_4867_49c0_bbb4_296600f06612.slice/crio-59f0647772decd7f97015d05638d4ac090d1889d1d20db29b77cd815182ccdfe WatchSource:0}: Error finding container 59f0647772decd7f97015d05638d4ac090d1889d1d20db29b77cd815182ccdfe: Status 404 returned error can't find the container with id 59f0647772decd7f97015d05638d4ac090d1889d1d20db29b77cd815182ccdfe Jul 18 02:30:00.363352 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.363321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg"] Jul 18 02:30:00.365824 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:30:00.365792 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e70bea5_8136_4902_9fbd_a1954f4a40c8.slice/crio-36649ba0fea07059a1135de9e0145c90ccb2a7e2955da0fef07c795689c1bcf4 WatchSource:0}: Error finding container 36649ba0fea07059a1135de9e0145c90ccb2a7e2955da0fef07c795689c1bcf4: Status 404 returned error can't find the container with id 36649ba0fea07059a1135de9e0145c90ccb2a7e2955da0fef07c795689c1bcf4 Jul 18 02:30:00.393407 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.393384 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2"] Jul 18 02:30:00.398823 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:30:00.398799 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667145e7_fe5e_475a_8ca9_70a7aa8e2b05.slice/crio-02a41e24cbea8cc5d82f0c3685830fa7fbbe01e990ecee7dbbaaf0544ccb36f6 WatchSource:0}: Error finding container 02a41e24cbea8cc5d82f0c3685830fa7fbbe01e990ecee7dbbaaf0544ccb36f6: Status 404 returned error can't find the container with id 02a41e24cbea8cc5d82f0c3685830fa7fbbe01e990ecee7dbbaaf0544ccb36f6 Jul 18 02:30:00.560902 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.560864 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg" event={"ID":"9e70bea5-8136-4902-9fbd-a1954f4a40c8","Type":"ContainerStarted","Data":"36649ba0fea07059a1135de9e0145c90ccb2a7e2955da0fef07c795689c1bcf4"} Jul 18 02:30:00.562087 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.562062 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5" event={"ID":"db919842-4867-49c0-bbb4-296600f06612","Type":"ContainerStarted","Data":"59f0647772decd7f97015d05638d4ac090d1889d1d20db29b77cd815182ccdfe"} Jul 18 02:30:00.563090 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:00.563070 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2" event={"ID":"667145e7-fe5e-475a-8ca9-70a7aa8e2b05","Type":"ContainerStarted","Data":"02a41e24cbea8cc5d82f0c3685830fa7fbbe01e990ecee7dbbaaf0544ccb36f6"} Jul 18 02:30:01.569113 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:01.569020 2582 generic.go:356] "Generic (PLEG): container finished" podID="9e70bea5-8136-4902-9fbd-a1954f4a40c8" containerID="892101724f2320affa1d93499b446a41aa4b8bbcde6a966ac6e4d3e5e4a4bae1" exitCode=0 Jul 18 02:30:01.569113 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:01.569091 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg" event={"ID":"9e70bea5-8136-4902-9fbd-a1954f4a40c8","Type":"ContainerDied","Data":"892101724f2320affa1d93499b446a41aa4b8bbcde6a966ac6e4d3e5e4a4bae1"} Jul 18 02:30:01.570572 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:01.570550 2582 generic.go:356] "Generic (PLEG): container finished" podID="db919842-4867-49c0-bbb4-296600f06612" containerID="56faade2a02c271d178a8f7601e0ff6359e4110afbd31f5f0c45a1ba2034ca65" exitCode=0 Jul 18 02:30:01.570682 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:01.570633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5" event={"ID":"db919842-4867-49c0-bbb4-296600f06612","Type":"ContainerDied","Data":"56faade2a02c271d178a8f7601e0ff6359e4110afbd31f5f0c45a1ba2034ca65"} Jul 18 02:30:01.572035 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:01.572017 2582 generic.go:356] "Generic (PLEG): container finished" podID="667145e7-fe5e-475a-8ca9-70a7aa8e2b05" containerID="72c33729bcfeec5d87151ec9baa0f28777eb582768f862aa8b3e923a5ad7f2a8" exitCode=0 Jul 18 02:30:01.572132 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:01.572065 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2" event={"ID":"667145e7-fe5e-475a-8ca9-70a7aa8e2b05","Type":"ContainerDied","Data":"72c33729bcfeec5d87151ec9baa0f28777eb582768f862aa8b3e923a5ad7f2a8"} Jul 18 02:30:02.728162 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:02.728120 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg" Jul 18 02:30:02.749249 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:02.749224 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5" Jul 18 02:30:02.753027 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:02.753010 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2" Jul 18 02:30:03.581787 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:03.581760 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg" Jul 18 02:30:03.581787 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:03.581768 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg" event={"ID":"9e70bea5-8136-4902-9fbd-a1954f4a40c8","Type":"ContainerDied","Data":"36649ba0fea07059a1135de9e0145c90ccb2a7e2955da0fef07c795689c1bcf4"} Jul 18 02:30:03.582034 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:03.581816 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36649ba0fea07059a1135de9e0145c90ccb2a7e2955da0fef07c795689c1bcf4" Jul 18 02:30:03.583073 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:03.583054 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5" Jul 18 02:30:03.583205 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:03.583093 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739030-tjbf5" event={"ID":"db919842-4867-49c0-bbb4-296600f06612","Type":"ContainerDied","Data":"59f0647772decd7f97015d05638d4ac090d1889d1d20db29b77cd815182ccdfe"} Jul 18 02:30:03.583205 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:03.583140 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59f0647772decd7f97015d05638d4ac090d1889d1d20db29b77cd815182ccdfe" Jul 18 02:30:03.584480 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:03.584448 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2" Jul 18 02:30:03.584580 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:03.584449 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2" event={"ID":"667145e7-fe5e-475a-8ca9-70a7aa8e2b05","Type":"ContainerDied","Data":"02a41e24cbea8cc5d82f0c3685830fa7fbbe01e990ecee7dbbaaf0544ccb36f6"} Jul 18 02:30:03.584631 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:03.584585 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02a41e24cbea8cc5d82f0c3685830fa7fbbe01e990ecee7dbbaaf0544ccb36f6" Jul 18 02:30:10.970227 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:10.970146 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:30:16.363599 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:16.363561 2582 scope.go:117] "RemoveContainer" containerID="63f79ca53cc2ed53689b3e1f0e3298d3c39e7abff4c827ee040ec8ef93c5b13c" Jul 18 02:30:34.163668 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:34.163630 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:30:47.859159 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:30:47.859126 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:31:16.383921 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:31:16.383891 2582 scope.go:117] "RemoveContainer" containerID="21c27972089efe6a86ad715185a42c93bdfe04eccc2ca9b5bcab56f90df68781" Jul 18 02:31:20.460857 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:31:20.460821 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:31:35.873496 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:31:35.873454 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:31:51.362929 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:31:51.362847 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:32:05.359810 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:05.359768 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:32:16.402823 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:16.402788 2582 scope.go:117] "RemoveContainer" containerID="8de8ff7c1a152669859e8719d9e8eae22c75c4fb0a7e45bd11921e0b8974c07d" Jul 18 02:32:16.411207 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:16.411181 2582 scope.go:117] "RemoveContainer" containerID="0a3977fc20789de5ca1c5af9b71f386106fc61b42fb23342cce17fdd5f55fb4c" Jul 18 02:32:19.605107 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605071 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8"] Jul 18 02:32:19.605484 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605419 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db919842-4867-49c0-bbb4-296600f06612" containerName="cleanup" Jul 18 02:32:19.605484 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605430 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db919842-4867-49c0-bbb4-296600f06612" containerName="cleanup" Jul 18 02:32:19.605484 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605439 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e70bea5-8136-4902-9fbd-a1954f4a40c8" containerName="cleanup" Jul 18 02:32:19.605484 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605445 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e70bea5-8136-4902-9fbd-a1954f4a40c8" containerName="cleanup" Jul 18 02:32:19.605484 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605467 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667145e7-fe5e-475a-8ca9-70a7aa8e2b05" containerName="cleanup" Jul 18 02:32:19.605484 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605473 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="667145e7-fe5e-475a-8ca9-70a7aa8e2b05" containerName="cleanup" Jul 18 02:32:19.605663 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605526 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="667145e7-fe5e-475a-8ca9-70a7aa8e2b05" containerName="cleanup" Jul 18 02:32:19.605663 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605535 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db919842-4867-49c0-bbb4-296600f06612" containerName="cleanup" Jul 18 02:32:19.605663 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.605543 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e70bea5-8136-4902-9fbd-a1954f4a40c8" containerName="cleanup" Jul 18 02:32:19.610102 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.610084 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" Jul 18 02:32:19.612860 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.612839 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vpdlz\"" Jul 18 02:32:19.623272 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.623245 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8"] Jul 18 02:32:19.729085 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.729049 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4acb8-d0dc-4503-ac27-a1b1783afe1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8\" (UID: \"2ca4acb8-d0dc-4503-ac27-a1b1783afe1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" Jul 18 02:32:19.829967 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.829931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4acb8-d0dc-4503-ac27-a1b1783afe1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8\" (UID: \"2ca4acb8-d0dc-4503-ac27-a1b1783afe1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" Jul 18 02:32:19.830562 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.830542 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4acb8-d0dc-4503-ac27-a1b1783afe1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8\" (UID: \"2ca4acb8-d0dc-4503-ac27-a1b1783afe1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" Jul 18 02:32:19.920071 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:19.919991 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" Jul 18 02:32:20.052091 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:20.052066 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8"] Jul 18 02:32:20.054235 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:32:20.054208 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca4acb8_d0dc_4503_ac27_a1b1783afe1e.slice/crio-a41eb8d951fc87ba48de3bd8319ff0ede31ed5ea45f9c746f1363267b94d78b4 WatchSource:0}: Error finding container a41eb8d951fc87ba48de3bd8319ff0ede31ed5ea45f9c746f1363267b94d78b4: Status 404 returned error can't find the container with id a41eb8d951fc87ba48de3bd8319ff0ede31ed5ea45f9c746f1363267b94d78b4 Jul 18 02:32:20.250058 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:20.250021 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" event={"ID":"2ca4acb8-d0dc-4503-ac27-a1b1783afe1e","Type":"ContainerStarted","Data":"a41eb8d951fc87ba48de3bd8319ff0ede31ed5ea45f9c746f1363267b94d78b4"} Jul 18 02:32:21.256055 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:21.256021 2582 generic.go:356] "Generic (PLEG): container finished" podID="2ca4acb8-d0dc-4503-ac27-a1b1783afe1e" containerID="a789c8defc1abb1e0a8ef749b83b2a1d83489898c719c09f92baa1c312224e45" exitCode=0 Jul 18 02:32:21.256450 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:21.256066 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" event={"ID":"2ca4acb8-d0dc-4503-ac27-a1b1783afe1e","Type":"ContainerDied","Data":"a789c8defc1abb1e0a8ef749b83b2a1d83489898c719c09f92baa1c312224e45"} Jul 18 02:32:22.386942 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:22.386914 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" Jul 18 02:32:22.452815 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:22.452785 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4acb8-d0dc-4503-ac27-a1b1783afe1e-maas-api-service-ca\") pod \"2ca4acb8-d0dc-4503-ac27-a1b1783afe1e\" (UID: \"2ca4acb8-d0dc-4503-ac27-a1b1783afe1e\") " Jul 18 02:32:22.453148 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:22.453125 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca4acb8-d0dc-4503-ac27-a1b1783afe1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ca4acb8-d0dc-4503-ac27-a1b1783afe1e" (UID: "2ca4acb8-d0dc-4503-ac27-a1b1783afe1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:32:22.554258 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:22.554184 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4acb8-d0dc-4503-ac27-a1b1783afe1e-maas-api-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:32:23.272260 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:23.272230 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" Jul 18 02:32:23.272260 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:23.272252 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8" event={"ID":"2ca4acb8-d0dc-4503-ac27-a1b1783afe1e","Type":"ContainerDied","Data":"a41eb8d951fc87ba48de3bd8319ff0ede31ed5ea45f9c746f1363267b94d78b4"} Jul 18 02:32:23.272469 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:23.272318 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a41eb8d951fc87ba48de3bd8319ff0ede31ed5ea45f9c746f1363267b94d78b4" Jul 18 02:32:30.169377 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.169344 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7"] Jul 18 02:32:30.169865 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.169632 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" podUID="fbbf07af-1216-4c15-8434-be033e89cd27" containerName="maas-api" containerID="cri-o://902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70" gracePeriod=30 Jul 18 02:32:30.204288 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.204258 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2"] Jul 18 02:32:30.206539 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.206515 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-deec4e-29739030-hfqt2"] Jul 18 02:32:30.620021 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.619996 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:32:30.732618 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.732529 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv8br\" (UniqueName: \"kubernetes.io/projected/fbbf07af-1216-4c15-8434-be033e89cd27-kube-api-access-xv8br\") pod \"fbbf07af-1216-4c15-8434-be033e89cd27\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " Jul 18 02:32:30.732618 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.732588 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls\") pod \"fbbf07af-1216-4c15-8434-be033e89cd27\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " Jul 18 02:32:30.732618 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.732622 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbf07af-1216-4c15-8434-be033e89cd27-openshift-service-ca\") pod \"fbbf07af-1216-4c15-8434-be033e89cd27\" (UID: \"fbbf07af-1216-4c15-8434-be033e89cd27\") " Jul 18 02:32:30.733070 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.733043 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbf07af-1216-4c15-8434-be033e89cd27-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fbbf07af-1216-4c15-8434-be033e89cd27" (UID: "fbbf07af-1216-4c15-8434-be033e89cd27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:32:30.735036 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.735006 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fbbf07af-1216-4c15-8434-be033e89cd27" (UID: "fbbf07af-1216-4c15-8434-be033e89cd27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:32:30.735036 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.735018 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbf07af-1216-4c15-8434-be033e89cd27-kube-api-access-xv8br" (OuterVolumeSpecName: "kube-api-access-xv8br") pod "fbbf07af-1216-4c15-8434-be033e89cd27" (UID: "fbbf07af-1216-4c15-8434-be033e89cd27"). InnerVolumeSpecName "kube-api-access-xv8br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:32:30.834232 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.834195 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv8br\" (UniqueName: \"kubernetes.io/projected/fbbf07af-1216-4c15-8434-be033e89cd27-kube-api-access-xv8br\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:32:30.834232 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.834226 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbf07af-1216-4c15-8434-be033e89cd27-maas-api-tls\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:32:30.834232 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:30.834235 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbf07af-1216-4c15-8434-be033e89cd27-openshift-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:32:31.310714 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:31.310676 2582 generic.go:356] "Generic (PLEG): container finished" podID="fbbf07af-1216-4c15-8434-be033e89cd27" containerID="902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70" exitCode=0 Jul 18 02:32:31.311384 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:31.310732 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" Jul 18 02:32:31.311384 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:31.310774 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" event={"ID":"fbbf07af-1216-4c15-8434-be033e89cd27","Type":"ContainerDied","Data":"902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70"} Jul 18 02:32:31.311384 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:31.310828 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7" event={"ID":"fbbf07af-1216-4c15-8434-be033e89cd27","Type":"ContainerDied","Data":"3f48b13ec416f409c4a7e707feac45323898e34d37048792a18e571a96f250c2"} Jul 18 02:32:31.311384 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:31.310860 2582 scope.go:117] "RemoveContainer" containerID="902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70" Jul 18 02:32:31.319800 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:31.319782 2582 scope.go:117] "RemoveContainer" containerID="902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70" Jul 18 02:32:31.320058 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:32:31.320033 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70\": container with ID starting with 902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70 not found: ID does not exist" containerID="902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70" Jul 18 02:32:31.320154 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:31.320062 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70"} err="failed to get container status \"902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70\": rpc error: code = NotFound desc = could not find container \"902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70\": container with ID starting with 902a0987d332a5d741e399da44dcef90b37a5a13fcbfe6c44520d8ba991b4a70 not found: ID does not exist" Jul 18 02:32:31.330978 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:31.330946 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7"] Jul 18 02:32:31.334664 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:31.334642 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-deec4e-5c6b89945f-4cqx7"] Jul 18 02:32:32.194617 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:32.194587 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667145e7-fe5e-475a-8ca9-70a7aa8e2b05" path="/var/lib/kubelet/pods/667145e7-fe5e-475a-8ca9-70a7aa8e2b05/volumes" Jul 18 02:32:32.194987 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:32.194973 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbbf07af-1216-4c15-8434-be033e89cd27" path="/var/lib/kubelet/pods/fbbf07af-1216-4c15-8434-be033e89cd27/volumes" Jul 18 02:32:33.460931 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:33.460893 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:32:34.688237 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:34.688202 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8"] Jul 18 02:32:34.690542 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:34.690514 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-deec4e-6l5v8"] Jul 18 02:32:36.194199 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:36.194167 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ca4acb8-d0dc-4503-ac27-a1b1783afe1e" path="/var/lib/kubelet/pods/2ca4acb8-d0dc-4503-ac27-a1b1783afe1e/volumes" Jul 18 02:32:41.964965 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.964935 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv"] Jul 18 02:32:41.965351 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.965274 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca4acb8-d0dc-4503-ac27-a1b1783afe1e" containerName="revoke-keys" Jul 18 02:32:41.965351 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.965285 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca4acb8-d0dc-4503-ac27-a1b1783afe1e" containerName="revoke-keys" Jul 18 02:32:41.965351 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.965302 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbf07af-1216-4c15-8434-be033e89cd27" containerName="maas-api" Jul 18 02:32:41.965351 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.965308 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbf07af-1216-4c15-8434-be033e89cd27" containerName="maas-api" Jul 18 02:32:41.965516 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.965375 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbf07af-1216-4c15-8434-be033e89cd27" containerName="maas-api" Jul 18 02:32:41.965516 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.965384 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca4acb8-d0dc-4503-ac27-a1b1783afe1e" containerName="revoke-keys" Jul 18 02:32:41.974717 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.974676 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" Jul 18 02:32:41.974960 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.974931 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv"] Jul 18 02:32:41.976994 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:41.976974 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vpdlz\"" Jul 18 02:32:42.138455 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:42.138422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d869d0b1-5842-4f73-837f-8db699f033b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv\" (UID: \"d869d0b1-5842-4f73-837f-8db699f033b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" Jul 18 02:32:42.239242 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:42.239157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d869d0b1-5842-4f73-837f-8db699f033b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv\" (UID: \"d869d0b1-5842-4f73-837f-8db699f033b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" Jul 18 02:32:42.239826 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:42.239807 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d869d0b1-5842-4f73-837f-8db699f033b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv\" (UID: \"d869d0b1-5842-4f73-837f-8db699f033b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" Jul 18 02:32:42.285583 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:42.285552 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" Jul 18 02:32:42.422976 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:42.422875 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv"] Jul 18 02:32:42.425144 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:32:42.425117 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd869d0b1_5842_4f73_837f_8db699f033b1.slice/crio-53613bed620e7f4bf45e0810680b0bcb07fb4bc9f3e36239bd641268df65e665 WatchSource:0}: Error finding container 53613bed620e7f4bf45e0810680b0bcb07fb4bc9f3e36239bd641268df65e665: Status 404 returned error can't find the container with id 53613bed620e7f4bf45e0810680b0bcb07fb4bc9f3e36239bd641268df65e665 Jul 18 02:32:43.369030 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:43.368931 2582 generic.go:356] "Generic (PLEG): container finished" podID="d869d0b1-5842-4f73-837f-8db699f033b1" containerID="b4f7ca8c69f5db0a18de48e3fb861a19c4a733e7014cb7d8c005971c4e98268c" exitCode=0 Jul 18 02:32:43.369030 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:43.368974 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" event={"ID":"d869d0b1-5842-4f73-837f-8db699f033b1","Type":"ContainerDied","Data":"b4f7ca8c69f5db0a18de48e3fb861a19c4a733e7014cb7d8c005971c4e98268c"} Jul 18 02:32:43.369030 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:43.369013 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" event={"ID":"d869d0b1-5842-4f73-837f-8db699f033b1","Type":"ContainerStarted","Data":"53613bed620e7f4bf45e0810680b0bcb07fb4bc9f3e36239bd641268df65e665"} Jul 18 02:32:44.526437 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:44.526411 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" Jul 18 02:32:44.659455 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:44.659356 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d869d0b1-5842-4f73-837f-8db699f033b1-maas-api-service-ca\") pod \"d869d0b1-5842-4f73-837f-8db699f033b1\" (UID: \"d869d0b1-5842-4f73-837f-8db699f033b1\") " Jul 18 02:32:44.659838 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:44.659806 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d869d0b1-5842-4f73-837f-8db699f033b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d869d0b1-5842-4f73-837f-8db699f033b1" (UID: "d869d0b1-5842-4f73-837f-8db699f033b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:32:44.760364 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:44.760330 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d869d0b1-5842-4f73-837f-8db699f033b1-maas-api-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:32:45.380392 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:45.380362 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" Jul 18 02:32:45.380532 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:45.380394 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv" event={"ID":"d869d0b1-5842-4f73-837f-8db699f033b1","Type":"ContainerDied","Data":"53613bed620e7f4bf45e0810680b0bcb07fb4bc9f3e36239bd641268df65e665"} Jul 18 02:32:45.380532 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:45.380439 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53613bed620e7f4bf45e0810680b0bcb07fb4bc9f3e36239bd641268df65e665" Jul 18 02:32:52.363978 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.363943 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4"] Jul 18 02:32:52.364486 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.364127 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" podUID="ee3fbdd1-de53-4379-8618-35e256f1e536" containerName="maas-api" containerID="cri-o://a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7" gracePeriod=30 Jul 18 02:32:52.394732 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.394692 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg"] Jul 18 02:32:52.396325 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.396304 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2a8451-29739030-w8ndg"] Jul 18 02:32:52.810953 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.810931 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:32:52.833139 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.833106 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3fbdd1-de53-4379-8618-35e256f1e536-maas-api-tls\") pod \"ee3fbdd1-de53-4379-8618-35e256f1e536\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " Jul 18 02:32:52.833304 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.833164 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3fbdd1-de53-4379-8618-35e256f1e536-openshift-service-ca\") pod \"ee3fbdd1-de53-4379-8618-35e256f1e536\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " Jul 18 02:32:52.833304 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.833203 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79sw6\" (UniqueName: \"kubernetes.io/projected/ee3fbdd1-de53-4379-8618-35e256f1e536-kube-api-access-79sw6\") pod \"ee3fbdd1-de53-4379-8618-35e256f1e536\" (UID: \"ee3fbdd1-de53-4379-8618-35e256f1e536\") " Jul 18 02:32:52.833541 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.833513 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3fbdd1-de53-4379-8618-35e256f1e536-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee3fbdd1-de53-4379-8618-35e256f1e536" (UID: "ee3fbdd1-de53-4379-8618-35e256f1e536"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:32:52.835397 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.835370 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3fbdd1-de53-4379-8618-35e256f1e536-kube-api-access-79sw6" (OuterVolumeSpecName: "kube-api-access-79sw6") pod "ee3fbdd1-de53-4379-8618-35e256f1e536" (UID: "ee3fbdd1-de53-4379-8618-35e256f1e536"). InnerVolumeSpecName "kube-api-access-79sw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:32:52.835549 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.835531 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3fbdd1-de53-4379-8618-35e256f1e536-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee3fbdd1-de53-4379-8618-35e256f1e536" (UID: "ee3fbdd1-de53-4379-8618-35e256f1e536"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:32:52.933975 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.933904 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3fbdd1-de53-4379-8618-35e256f1e536-maas-api-tls\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:32:52.933975 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.933926 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3fbdd1-de53-4379-8618-35e256f1e536-openshift-service-ca\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:32:52.933975 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:52.933937 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79sw6\" (UniqueName: \"kubernetes.io/projected/ee3fbdd1-de53-4379-8618-35e256f1e536-kube-api-access-79sw6\") on node \"ip-10-0-135-157.ec2.internal\" DevicePath \"\"" Jul 18 02:32:53.421293 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:53.421252 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee3fbdd1-de53-4379-8618-35e256f1e536" containerID="a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7" exitCode=0 Jul 18 02:32:53.421922 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:53.421306 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" Jul 18 02:32:53.421922 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:53.421339 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" event={"ID":"ee3fbdd1-de53-4379-8618-35e256f1e536","Type":"ContainerDied","Data":"a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7"} Jul 18 02:32:53.421922 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:53.421407 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4" event={"ID":"ee3fbdd1-de53-4379-8618-35e256f1e536","Type":"ContainerDied","Data":"debe45db22faab68834a609c0bb719ce5f5b298ab11108e28f8849ae5656123f"} Jul 18 02:32:53.421922 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:53.421442 2582 scope.go:117] "RemoveContainer" containerID="a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7" Jul 18 02:32:53.438410 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:53.438387 2582 scope.go:117] "RemoveContainer" containerID="a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7" Jul 18 02:32:53.438731 ip-10-0-135-157 kubenswrapper[2582]: E0718 02:32:53.438707 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7\": container with ID starting with a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7 not found: ID does not exist" containerID="a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7" Jul 18 02:32:53.438837 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:53.438756 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7"} err="failed to get container status \"a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7\": rpc error: code = NotFound desc = could not find container \"a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7\": container with ID starting with a3d5b9f47cabe3433cbcc363f60c328cc877a25f4e36ff4b897e736cc66083a7 not found: ID does not exist" Jul 18 02:32:53.447718 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:53.447690 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4"] Jul 18 02:32:53.451326 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:53.451299 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2a8451-67ff4b759f-44th4"] Jul 18 02:32:54.194639 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:54.194606 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e70bea5-8136-4902-9fbd-a1954f4a40c8" path="/var/lib/kubelet/pods/9e70bea5-8136-4902-9fbd-a1954f4a40c8/volumes" Jul 18 02:32:54.195002 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:54.194988 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee3fbdd1-de53-4379-8618-35e256f1e536" path="/var/lib/kubelet/pods/ee3fbdd1-de53-4379-8618-35e256f1e536/volumes" Jul 18 02:32:55.158186 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:55.158152 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrznl"] Jul 18 02:32:57.037400 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:57.037364 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv"] Jul 18 02:32:57.040876 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:57.040852 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2a8451-lwztv"] Jul 18 02:32:58.194781 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:32:58.194720 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d869d0b1-5842-4f73-837f-8db699f033b1" path="/var/lib/kubelet/pods/d869d0b1-5842-4f73-837f-8db699f033b1/volumes" Jul 18 02:33:10.911843 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:10.911771 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b7456dd48-59hvp_740e4863-0fa0-459a-a89e-1fe74ec76b16/copy-manifests/0.log" Jul 18 02:33:10.918819 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:10.918794 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b7456dd48-59hvp_740e4863-0fa0-459a-a89e-1fe74ec76b16/manager/0.log" Jul 18 02:33:11.719156 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:11.719128 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29739030-tjbf5_db919842-4867-49c0-bbb4-296600f06612/cleanup/0.log" Jul 18 02:33:13.838534 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:13.838496 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jrznl_e38b18d2-a337-4407-b215-a47a5ac90eed/limitador/0.log" Jul 18 02:33:14.424245 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:14.424209 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-55w5d_0f69470a-71fe-4730-8bac-d716db010398/discovery/0.log" Jul 18 02:33:15.922471 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:15.922434 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6cvz6_9d30c8f9-fe02-46dc-9f96-ccffd94d9cbb/main/0.log" Jul 18 02:33:16.039996 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:16.039961 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-jpmdm_aec05950-dffb-40c7-a4de-0bedade71d44/main/0.log" Jul 18 02:33:23.887425 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:23.887391 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ndt4g_55d05391-7c5e-416f-a050-ed1b32053280/global-pull-secret-syncer/0.log" Jul 18 02:33:24.011657 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:24.011629 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n6gnr_f75d2da2-02ba-40e4-8a9d-efe762a83e51/konnectivity-agent/0.log" Jul 18 02:33:24.103400 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:24.103369 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-157.ec2.internal_38b08105c0224d562f53bf7954a3df13/haproxy/0.log" Jul 18 02:33:29.958985 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:29.958951 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jrznl_e38b18d2-a337-4407-b215-a47a5ac90eed/limitador/0.log" Jul 18 02:33:31.716591 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.716522 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a6eea2-53c2-4861-bfe3-884ffbbdbaef/alertmanager/0.log" Jul 18 02:33:31.740011 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.739982 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a6eea2-53c2-4861-bfe3-884ffbbdbaef/config-reloader/0.log" Jul 18 02:33:31.763362 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.763342 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a6eea2-53c2-4861-bfe3-884ffbbdbaef/kube-rbac-proxy-web/0.log" Jul 18 02:33:31.786403 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.786380 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a6eea2-53c2-4861-bfe3-884ffbbdbaef/kube-rbac-proxy/0.log" Jul 18 02:33:31.809446 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.809419 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a6eea2-53c2-4861-bfe3-884ffbbdbaef/kube-rbac-proxy-metric/0.log" Jul 18 02:33:31.832373 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.832345 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a6eea2-53c2-4861-bfe3-884ffbbdbaef/prom-label-proxy/0.log" Jul 18 02:33:31.856768 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.856731 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a6eea2-53c2-4861-bfe3-884ffbbdbaef/init-config-reloader/0.log" Jul 18 02:33:31.927882 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.927855 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-x9dzv_7502a389-1c73-417f-9f71-cf494086c9f1/kube-state-metrics/0.log" Jul 18 02:33:31.947084 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.947052 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-x9dzv_7502a389-1c73-417f-9f71-cf494086c9f1/kube-rbac-proxy-main/0.log" Jul 18 02:33:31.971796 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.971704 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-x9dzv_7502a389-1c73-417f-9f71-cf494086c9f1/kube-rbac-proxy-self/0.log" Jul 18 02:33:31.999444 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:31.999420 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bb6bf6858-gp2p4_c681235c-786d-4e6b-bd53-923cac202780/metrics-server/0.log" Jul 18 02:33:32.232938 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.232864 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvgnb_0a7251de-e1da-4896-913d-6c002dbfe03f/node-exporter/0.log" Jul 18 02:33:32.263377 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.263352 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvgnb_0a7251de-e1da-4896-913d-6c002dbfe03f/kube-rbac-proxy/0.log" Jul 18 02:33:32.282361 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.282332 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvgnb_0a7251de-e1da-4896-913d-6c002dbfe03f/init-textfile/0.log" Jul 18 02:33:32.309828 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.309805 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-7r6xz_ed19f185-adee-4f8a-a293-937d1dcd0df4/kube-rbac-proxy-main/0.log" Jul 18 02:33:32.330869 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.330846 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-7r6xz_ed19f185-adee-4f8a-a293-937d1dcd0df4/kube-rbac-proxy-self/0.log" Jul 18 02:33:32.352456 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.352433 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-7r6xz_ed19f185-adee-4f8a-a293-937d1dcd0df4/openshift-state-metrics/0.log" Jul 18 02:33:32.556795 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.556693 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6nvkr_91ceef4e-58cd-4918-a498-0db69ed5fe66/prometheus-operator/0.log" Jul 18 02:33:32.576663 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.576639 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6nvkr_91ceef4e-58cd-4918-a498-0db69ed5fe66/kube-rbac-proxy/0.log" Jul 18 02:33:32.605964 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.605937 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-46j8c_eda6f788-88a2-4aed-a1c1-bdabc978bfee/prometheus-operator-admission-webhook/0.log" Jul 18 02:33:32.641559 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.641535 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66b85cb67-z74gv_2a7ac8b0-b459-4975-9afc-ed9d9d7a8158/telemeter-client/0.log" Jul 18 02:33:32.664792 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.664768 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66b85cb67-z74gv_2a7ac8b0-b459-4975-9afc-ed9d9d7a8158/reload/0.log" Jul 18 02:33:32.685681 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:32.685658 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66b85cb67-z74gv_2a7ac8b0-b459-4975-9afc-ed9d9d7a8158/kube-rbac-proxy/0.log" Jul 18 02:33:33.959895 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.959859 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp"] Jul 18 02:33:33.960380 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.960362 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee3fbdd1-de53-4379-8618-35e256f1e536" containerName="maas-api" Jul 18 02:33:33.960451 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.960384 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee3fbdd1-de53-4379-8618-35e256f1e536" containerName="maas-api" Jul 18 02:33:33.960451 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.960407 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d869d0b1-5842-4f73-837f-8db699f033b1" containerName="revoke-keys" Jul 18 02:33:33.960451 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.960416 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d869d0b1-5842-4f73-837f-8db699f033b1" containerName="revoke-keys" Jul 18 02:33:33.960596 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.960497 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d869d0b1-5842-4f73-837f-8db699f033b1" containerName="revoke-keys" Jul 18 02:33:33.960596 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.960517 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee3fbdd1-de53-4379-8618-35e256f1e536" containerName="maas-api" Jul 18 02:33:33.963672 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.963649 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:33.965928 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.965905 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9d7x6\"/\"openshift-service-ca.crt\"" Jul 18 02:33:33.966041 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.965956 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9d7x6\"/\"default-dockercfg-6b44b\"" Jul 18 02:33:33.966809 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.966789 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9d7x6\"/\"kube-root-ca.crt\"" Jul 18 02:33:33.972294 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:33.972267 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp"] Jul 18 02:33:34.098301 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.098268 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-proc\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.098301 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.098305 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-lib-modules\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.098550 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.098338 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-sys\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.098550 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.098422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fw86\" (UniqueName: \"kubernetes.io/projected/c8610519-ea72-4c35-9887-6c438b3a8387-kube-api-access-7fw86\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.098550 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.098452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-podres\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.199079 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.199047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-proc\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.199079 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.199079 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-lib-modules\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.199302 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.199102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-sys\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.199302 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.199135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fw86\" (UniqueName: \"kubernetes.io/projected/c8610519-ea72-4c35-9887-6c438b3a8387-kube-api-access-7fw86\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.199302 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.199153 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-podres\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.199302 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.199175 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-proc\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.199302 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.199199 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-sys\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.199302 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.199237 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-lib-modules\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.199302 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.199268 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8610519-ea72-4c35-9887-6c438b3a8387-podres\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.206718 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.206690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fw86\" (UniqueName: \"kubernetes.io/projected/c8610519-ea72-4c35-9887-6c438b3a8387-kube-api-access-7fw86\") pod \"perf-node-gather-daemonset-srwfp\" (UID: \"c8610519-ea72-4c35-9887-6c438b3a8387\") " pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.275172 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.275072 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.409388 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.409356 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp"] Jul 18 02:33:34.415453 ip-10-0-135-157 kubenswrapper[2582]: W0718 02:33:34.413052 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc8610519_ea72_4c35_9887_6c438b3a8387.slice/crio-1ef61d665b91f287315ff6a53124efc0246cb23481bb9afcbb756e4fa49332bd WatchSource:0}: Error finding container 1ef61d665b91f287315ff6a53124efc0246cb23481bb9afcbb756e4fa49332bd: Status 404 returned error can't find the container with id 1ef61d665b91f287315ff6a53124efc0246cb23481bb9afcbb756e4fa49332bd Jul 18 02:33:34.627763 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.627700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" event={"ID":"c8610519-ea72-4c35-9887-6c438b3a8387","Type":"ContainerStarted","Data":"e85e9e848df62207da6b3f69a99cf2af9b52204f5f2e5723b27efe753c7b66d3"} Jul 18 02:33:34.627933 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.627771 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" event={"ID":"c8610519-ea72-4c35-9887-6c438b3a8387","Type":"ContainerStarted","Data":"1ef61d665b91f287315ff6a53124efc0246cb23481bb9afcbb756e4fa49332bd"} Jul 18 02:33:34.627933 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.627799 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:34.645284 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:34.645245 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" podStartSLOduration=1.645229328 podStartE2EDuration="1.645229328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:33:34.644579637 +0000 UTC m=+2718.975820630" watchObservedRunningTime="2026-07-18 02:33:34.645229328 +0000 UTC m=+2718.976470309" Jul 18 02:33:35.127230 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:35.127197 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-rkcvx_ce19db55-e204-4fa9-9e60-6eaca36ca929/download-server/0.log" Jul 18 02:33:36.750767 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:36.750723 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bfjfq_0d33b730-8ecf-43aa-a72e-b4d9f0ad6737/dns/0.log" Jul 18 02:33:36.773956 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:36.773928 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bfjfq_0d33b730-8ecf-43aa-a72e-b4d9f0ad6737/kube-rbac-proxy/0.log" Jul 18 02:33:36.890727 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:36.890700 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-22pm6_b4d10c59-9d5a-476b-8a8f-091d79787977/dns-node-resolver/0.log" Jul 18 02:33:37.619333 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:37.619308 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ghg58_3a3ad0e3-9c72-43c5-8d9f-b45eb96f42a8/node-ca/0.log" Jul 18 02:33:38.737369 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:38.737340 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-55w5d_0f69470a-71fe-4730-8bac-d716db010398/discovery/0.log" Jul 18 02:33:39.641546 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:39.641513 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lcdmx_c04b51a1-a506-4586-a8da-b11bc378a4da/serve-healthcheck-canary/0.log" Jul 18 02:33:40.155961 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:40.155936 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5tjcm_aadd2bf8-b1fa-4a7f-bfa0-468ce296e214/kube-rbac-proxy/0.log" Jul 18 02:33:40.180460 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:40.180437 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5tjcm_aadd2bf8-b1fa-4a7f-bfa0-468ce296e214/exporter/0.log" Jul 18 02:33:40.200732 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:40.200708 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5tjcm_aadd2bf8-b1fa-4a7f-bfa0-468ce296e214/extractor/0.log" Jul 18 02:33:40.643439 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:40.643412 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9d7x6/perf-node-gather-daemonset-srwfp" Jul 18 02:33:43.111547 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:43.111517 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b7456dd48-59hvp_740e4863-0fa0-459a-a89e-1fe74ec76b16/manager/0.log" Jul 18 02:33:43.136267 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:43.136235 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b7456dd48-59hvp_740e4863-0fa0-459a-a89e-1fe74ec76b16/copy-manifests/0.log" Jul 18 02:33:44.596249 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:44.596174 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c55bb847-xtnpj_51e9b6eb-cbbe-469f-b702-d26915483616/manager/0.log" Jul 18 02:33:44.624256 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:44.624231 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-2vsv4_b933d8b2-77d2-40f0-81aa-7cc229fd2599/openshift-lws-operator/0.log" Jul 18 02:33:44.849581 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:44.849555 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-55w5d_0f69470a-71fe-4730-8bac-d716db010398/discovery/0.log" Jul 18 02:33:46.322065 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:46.322029 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-55w5d_0f69470a-71fe-4730-8bac-d716db010398/discovery/0.log" Jul 18 02:33:50.792469 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:50.792435 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-49dkv_2d052cd5-4ee5-4cdd-bd68-3ddeeb6f62d0/kube-multus/0.log" Jul 18 02:33:51.136118 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:51.136084 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwdx9_440cebb3-367e-4f50-9418-46b7182e7520/kube-multus-additional-cni-plugins/0.log" Jul 18 02:33:51.155391 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:51.155370 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwdx9_440cebb3-367e-4f50-9418-46b7182e7520/egress-router-binary-copy/0.log" Jul 18 02:33:51.174255 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:51.174229 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwdx9_440cebb3-367e-4f50-9418-46b7182e7520/cni-plugins/0.log" Jul 18 02:33:51.193327 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:51.193297 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwdx9_440cebb3-367e-4f50-9418-46b7182e7520/bond-cni-plugin/0.log" Jul 18 02:33:51.212925 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:51.212899 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwdx9_440cebb3-367e-4f50-9418-46b7182e7520/routeoverride-cni/0.log" Jul 18 02:33:51.234460 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:51.234435 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwdx9_440cebb3-367e-4f50-9418-46b7182e7520/whereabouts-cni-bincopy/0.log" Jul 18 02:33:51.254362 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:51.254339 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwdx9_440cebb3-367e-4f50-9418-46b7182e7520/whereabouts-cni/0.log" Jul 18 02:33:51.419446 ip-10-0-135-157 kubenswrapper[2582]: I0718 02:33:51.419368 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v4r5t_6d8e552e-32ad-4c02-8e89-bc9d00d093e5/network-metrics-daemon/0.log"