Jul 28 17:34:34.911456 ip-10-0-139-157 systemd[1]: Starting Kubernetes Kubelet... Jul 28 17:34:35.291623 ip-10-0-139-157 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:34:35.291623 ip-10-0-139-157 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 17:34:35.291623 ip-10-0-139-157 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:34:35.291623 ip-10-0-139-157 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 17:34:35.291623 ip-10-0-139-157 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:34:35.293285 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.293195 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 17:34:35.296774 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296759 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:34:35.296774 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296775 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296779 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296782 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296785 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296788 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296792 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296794 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296797 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296800 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296803 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296806 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296809 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296812 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296815 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296818 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296820 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296823 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296828 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296832 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:34:35.296835 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296835 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296838 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296841 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296844 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296847 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296850 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296853 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296856 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296858 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296861 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296863 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296866 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296868 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296871 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296873 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296876 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296879 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296881 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296884 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296886 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:34:35.297308 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296889 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296891 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296894 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296897 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296899 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296902 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296905 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296907 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296910 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296912 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296915 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296917 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296920 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296922 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296926 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296928 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296932 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296936 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296939 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:34:35.297787 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296943 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296947 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296950 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296953 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296956 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296962 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296965 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296968 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296971 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296974 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296977 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296980 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296982 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296985 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296988 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296991 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296993 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296996 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.296998 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297001 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:34:35.298278 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297003 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297006 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297008 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297011 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297014 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297017 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297019 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297022 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297025 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297028 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297032 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297035 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297037 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297040 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297043 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297045 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297048 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297050 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297053 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297056 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:34:35.298789 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297758 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297766 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297769 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297772 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297775 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297777 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297780 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297782 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297785 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297788 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297790 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297793 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297795 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297798 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297801 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297803 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297807 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297809 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297812 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297815 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:34:35.299304 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297818 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297821 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297824 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297826 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297829 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297831 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297834 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297836 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297839 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297841 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297844 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297848 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297851 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297854 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297857 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297860 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297862 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297865 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297867 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:34:35.299805 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297870 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297872 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297875 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297877 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297880 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297882 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297884 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297887 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297889 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297892 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297894 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297897 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297902 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297905 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297907 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297910 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297912 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297915 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297917 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297920 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:34:35.300301 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297923 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297925 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297928 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297930 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297933 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297936 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297938 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297941 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297944 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297946 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297949 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297951 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297954 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297957 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297959 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297961 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297964 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297966 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297969 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297972 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:34:35.300796 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297974 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297979 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297982 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297986 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297989 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297992 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297995 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.297997 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298000 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298002 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298005 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298007 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298010 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298012 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298015 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298018 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298020 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298023 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298026 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:34:35.301298 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298028 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298414 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298419 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298422 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298424 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298427 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298430 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298432 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298435 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298437 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298440 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298442 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298445 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298447 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298452 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298455 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298459 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298463 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298466 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298469 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:34:35.301769 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298472 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298475 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298478 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298480 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298483 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298486 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298488 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298491 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298493 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298496 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298498 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298501 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298504 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298506 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298509 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298511 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298515 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298517 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298520 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:34:35.302307 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298524 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298527 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298530 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298533 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298536 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298539 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298541 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298544 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298546 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298549 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298552 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298554 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298556 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298559 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298562 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298564 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298567 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298569 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298572 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298574 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:34:35.302788 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298577 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298579 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298582 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298584 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298587 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298590 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298593 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298596 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298598 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298605 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298608 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298610 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298613 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298615 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298618 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298621 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298624 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298626 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298629 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298632 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:34:35.303306 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298634 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298637 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298639 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298642 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298645 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298647 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298650 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298652 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298655 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298657 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298660 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298662 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298665 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298667 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298670 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298673 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298675 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298677 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298681 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298684 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:34:35.303830 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.298687 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298761 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298767 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298773 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298778 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298782 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298786 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298790 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298794 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298797 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298800 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298804 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298808 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298811 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298814 2576 flags.go:64] FLAG: --cgroup-root="" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298818 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298821 2576 flags.go:64] FLAG: --client-ca-file="" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298824 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298826 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298830 2576 flags.go:64] FLAG: --cluster-domain="" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298832 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298835 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298838 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298842 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 17:34:35.304324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298845 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298849 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298852 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298855 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298858 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298861 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298864 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298868 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298886 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298891 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298895 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298898 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298901 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298905 2576 flags.go:64] FLAG: --enable-server="true" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298908 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298912 2576 flags.go:64] FLAG: --event-burst="100" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298915 2576 flags.go:64] FLAG: --event-qps="50" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298918 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298921 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298923 2576 flags.go:64] FLAG: --eviction-hard="" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298927 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298930 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298933 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298936 2576 flags.go:64] FLAG: --eviction-soft="" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298939 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 17:34:35.304948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298942 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298944 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298947 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298950 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298953 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298956 2576 flags.go:64] FLAG: --feature-gates="" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298960 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298963 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298966 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298969 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298972 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298975 2576 flags.go:64] FLAG: --help="false" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298978 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298981 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298983 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298987 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298991 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298994 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.298997 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299000 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299003 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299006 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299009 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299012 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 17:34:35.305710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299015 2576 flags.go:64] FLAG: --kube-reserved="" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299018 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299020 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299024 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299026 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299029 2576 flags.go:64] FLAG: --lock-file="" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299032 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299035 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299037 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299043 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299046 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299049 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299052 2576 flags.go:64] FLAG: --logging-format="text" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299055 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299058 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299061 2576 flags.go:64] FLAG: --manifest-url="" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299073 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299078 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299081 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299085 2576 flags.go:64] FLAG: --max-pods="110" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299088 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299091 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299094 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299097 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299101 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 17:34:35.306347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299104 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299107 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299113 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299117 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299120 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299123 2576 flags.go:64] FLAG: --pod-cidr="" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299126 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299131 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299134 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299137 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299139 2576 flags.go:64] FLAG: --port="10250" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299142 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299145 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-049796952e1e4786c" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299148 2576 flags.go:64] FLAG: --qos-reserved="" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299151 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299154 2576 flags.go:64] FLAG: --register-node="true" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299157 2576 flags.go:64] FLAG: --register-with-taints="" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299161 2576 flags.go:64] FLAG: --registry-burst="10" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299164 2576 flags.go:64] FLAG: --registry-qps="5" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299167 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299181 2576 flags.go:64] FLAG: --reserved-memory="" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299185 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299189 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299192 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299195 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 17:34:35.306959 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299198 2576 flags.go:64] FLAG: --runonce="false" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299201 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299204 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299207 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299210 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299212 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299215 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299219 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299222 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299225 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299228 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299231 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299234 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299237 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299239 2576 flags.go:64] FLAG: --system-cgroups="" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299242 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299248 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299250 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299253 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299256 2576 flags.go:64] FLAG: --tls-min-version="" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299259 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299262 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299264 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299267 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299270 2576 flags.go:64] FLAG: --v="2" Jul 28 17:34:35.307868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299274 2576 flags.go:64] FLAG: --version="false" Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299278 2576 flags.go:64] FLAG: --vmodule="" Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299281 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.299285 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299401 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299405 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299408 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299410 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299413 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299416 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299419 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299421 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299424 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299428 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299431 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299435 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299438 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299440 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299443 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299446 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299448 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:34:35.308714 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299451 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299454 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299456 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299459 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299461 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299464 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299467 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299469 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299472 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299474 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299477 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299479 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299484 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299487 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299489 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299492 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299494 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299497 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299499 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299502 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:34:35.309289 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299504 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299508 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299511 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299514 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299519 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299521 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299524 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299527 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299530 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299533 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299536 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299538 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299542 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299545 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299548 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299550 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299553 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299556 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299558 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299561 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:34:35.309785 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299563 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299566 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299569 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299571 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299575 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299578 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299580 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299583 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299586 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299588 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299591 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299593 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299596 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299598 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299601 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299603 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299607 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299610 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299613 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:34:35.310295 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299615 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299618 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299621 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299624 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299626 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299629 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299632 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299634 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299637 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299640 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299642 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299645 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299647 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299650 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299652 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299655 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299657 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299661 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299664 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:34:35.310759 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299666 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299669 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299671 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.299675 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.300347 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.308975 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.308993 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.309013 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.309020 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309078 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309084 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309088 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309091 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309095 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:34:35.311241 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309098 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309101 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309104 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309106 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309109 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309112 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309115 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309118 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309120 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309123 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309127 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309130 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309133 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309136 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309139 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309142 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309145 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309148 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309151 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:34:35.311606 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309154 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309157 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309160 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309163 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309166 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309187 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309191 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309194 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309197 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309200 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309203 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309205 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309208 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309211 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309213 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309216 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309219 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309222 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309224 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309227 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:34:35.312058 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309230 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309233 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309235 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309238 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309241 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309243 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309246 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309248 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309251 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309253 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309256 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309258 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309261 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309264 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309267 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309271 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309275 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309278 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309280 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:34:35.312561 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309283 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309286 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309288 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309291 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309294 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309296 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309299 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309302 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309305 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309308 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309310 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309313 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309317 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309320 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309323 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309326 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309329 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309331 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309334 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:34:35.313024 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309336 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309339 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309341 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309344 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309347 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309349 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309352 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309354 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309357 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309360 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309363 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309365 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309368 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309370 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309373 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309376 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:34:35.313652 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309378 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.309383 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309502 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309509 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309513 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309516 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309519 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309522 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309525 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309528 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309531 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309533 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309536 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309538 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309541 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:34:35.314038 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309544 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309547 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309549 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309552 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309555 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309559 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309561 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309564 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309567 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309569 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309572 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309575 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309578 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309580 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309583 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309586 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309588 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309592 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309595 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:34:35.314424 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309597 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309600 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309603 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309606 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309608 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309611 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309613 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309616 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309619 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309621 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309624 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309626 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309629 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309632 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309635 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309637 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309640 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309642 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309645 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309647 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:34:35.314984 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309650 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309652 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309655 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309657 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309660 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309663 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309666 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309668 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309672 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309676 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309679 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309681 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309684 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309686 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309689 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309692 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309694 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309697 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309699 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309702 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:34:35.315500 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309704 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309706 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309709 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309711 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309714 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309717 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309720 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309722 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309725 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309727 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309730 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309732 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309735 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309737 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309740 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309742 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309745 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309748 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309751 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:34:35.316099 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309753 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309756 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309759 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309761 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309764 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309766 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309769 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:35.309771 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.309776 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.309899 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.313278 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.314638 2576 server.go:1019] "Starting client certificate rotation" Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.314758 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:34:35.316844 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.314800 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:34:35.317386 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.317372 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:34:35.318603 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.318576 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:34:35.327683 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.327668 2576 log.go:25] "Validated CRI v1 runtime API" Jul 28 17:34:35.329799 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.329788 2576 log.go:25] "Validated CRI v1 image API" Jul 28 17:34:35.331340 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.331325 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 17:34:35.338390 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.338371 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e1c4e03d-2c4e-4b5b-ba07-75dde2cdf844:/dev/nvme0n1p4 fb14b768-2146-48cd-a1ef-0f00ffede508:/dev/nvme0n1p3] Jul 28 17:34:35.338433 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.338391 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 17:34:35.343971 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.343858 2576 manager.go:216] Machine: {Timestamp:2026-07-28 17:34:35.341811929 +0000 UTC m=+0.314319472 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3139115 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2979e1f9e0b4e9c871335f244abc0b SystemUUID:ec2979e1-f9e0-b4e9-c871-335f244abc0b BootID:e2cc21df-12d4-4e1a-ad6c-08fab14e6328 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0e:72:46:25:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0e:72:46:25:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:d8:5e:86:11:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 17:34:35.343971 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.343968 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 17:34:35.344091 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.344080 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 17:34:35.348148 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.347815 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 17:34:35.348328 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.348147 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-157.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 17:34:35.349462 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.349450 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 17:34:35.349498 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.349465 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 17:34:35.349498 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.349472 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:34:35.350067 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.350057 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:34:35.350542 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.350522 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:34:35.351261 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.351250 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 17:34:35.353031 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.353020 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:34:35.353147 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.353137 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 17:34:35.353189 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.353167 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 28 17:34:35.353217 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.353193 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 17:34:35.353217 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.353208 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 17:34:35.353217 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.353217 2576 kubelet.go:398] "Adding apiserver pod source" Jul 28 17:34:35.353309 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.353226 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 17:34:35.353628 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.353616 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:34:35.353663 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.353634 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:34:35.355902 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.355880 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 17:34:35.358000 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.357987 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 17:34:35.358058 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.358017 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 17:34:35.359415 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359404 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 17:34:35.359461 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359419 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 17:34:35.359461 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359425 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 17:34:35.359461 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359431 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 17:34:35.359461 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359440 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 17:34:35.359461 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359445 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 17:34:35.359461 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359454 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 17:34:35.359461 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359459 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 17:34:35.359461 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359465 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 17:34:35.359787 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359471 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 17:34:35.359787 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359479 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 17:34:35.359787 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359516 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 17:34:35.359787 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359542 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 17:34:35.359787 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.359547 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 17:34:35.362667 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.362650 2576 server.go:1268] "Started kubelet" Jul 28 17:34:35.362812 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.362757 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 17:34:35.362866 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.362835 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 17:34:35.363263 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.363101 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 17:34:35.363615 ip-10-0-139-157 systemd[1]: Started Kubernetes Kubelet. Jul 28 17:34:35.364425 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.364407 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 17:34:35.365140 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.365128 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 28 17:34:35.367961 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.367933 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 17:34:35.368051 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.367969 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-157.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 17:34:35.368137 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.368114 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 17:34:35.368502 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.368485 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 17:34:35.368869 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.367932 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-157.ec2.internal.18c684946fad0c1e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-157.ec2.internal,UID:ip-10-0-139-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-157.ec2.internal,},FirstTimestamp:2026-07-28 17:34:35.362618398 +0000 UTC m=+0.335125940,LastTimestamp:2026-07-28 17:34:35.362618398 +0000 UTC m=+0.335125940,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-157.ec2.internal,}" Jul 28 17:34:35.372061 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.371521 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 17:34:35.372061 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.371564 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 17:34:35.372178 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.372073 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:35.372253 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372193 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 17:34:35.372253 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372195 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 17:34:35.372253 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372217 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 17:34:35.372405 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372313 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 28 17:34:35.372405 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372324 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 28 17:34:35.372539 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372518 2576 factory.go:55] Registering systemd factory Jul 28 17:34:35.372539 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372540 2576 factory.go:223] Registration of the systemd container factory successfully Jul 28 17:34:35.372829 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372816 2576 factory.go:153] Registering CRI-O factory Jul 28 17:34:35.372903 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372833 2576 factory.go:223] Registration of the crio container factory successfully Jul 28 17:34:35.372903 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372891 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 17:34:35.373026 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372916 2576 factory.go:103] Registering Raw factory Jul 28 17:34:35.373026 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.372931 2576 manager.go:1196] Started watching for new ooms in manager Jul 28 17:34:35.373621 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.373607 2576 manager.go:349] Starting recovery of all containers Jul 28 17:34:35.374309 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.374283 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 17:34:35.374434 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.374383 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 17:34:35.385366 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.385339 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5lz6" Jul 28 17:34:35.386098 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.386079 2576 manager.go:354] Recovery completed Jul 28 17:34:35.387608 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.387589 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 28 17:34:35.390971 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.390954 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:34:35.394360 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.394342 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:34:35.394443 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.394375 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:34:35.394443 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.394387 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:34:35.395152 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.395135 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 17:34:35.395152 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.395143 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:34:35.395152 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.395148 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 17:34:35.395351 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.395166 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:34:35.395597 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.395580 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5lz6" Jul 28 17:34:35.395994 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.395976 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:34:35.396049 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.396007 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:34:35.396049 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.396021 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:34:35.397324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.397303 2576 policy_none.go:49] "None policy: Start" Jul 28 17:34:35.397414 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.397329 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 17:34:35.397414 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.397342 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 17:34:35.398582 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.398571 2576 policy_none.go:47] "Start" Jul 28 17:34:35.437233 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.436998 2576 manager.go:337] "Starting Device Plugin manager" Jul 28 17:34:35.437393 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.437327 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 17:34:35.437393 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.437343 2576 server.go:87] "Starting device plugin registration server" Jul 28 17:34:35.437635 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.437623 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 17:34:35.437692 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.437637 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 17:34:35.437796 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.437769 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 17:34:35.437898 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.437885 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 17:34:35.437898 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.437899 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 17:34:35.438555 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.438534 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 17:34:35.438647 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.438586 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:35.538408 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.538372 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:34:35.539776 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.539534 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:34:35.539776 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.539573 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:34:35.539776 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.539589 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:34:35.539776 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.539632 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.549530 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.549452 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.549530 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.549484 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-157.ec2.internal\": node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:35.566413 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.566380 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:35.588100 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.588055 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 17:34:35.589405 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.589386 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 17:34:35.589485 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.589430 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 17:34:35.589485 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.589458 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 17:34:35.589579 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.589497 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 17:34:35.592405 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.592379 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:34:35.667854 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.667808 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:35.690970 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.690936 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal"] Jul 28 17:34:35.691065 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.691049 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:34:35.692247 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.692220 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:34:35.692361 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.692253 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:34:35.692361 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.692264 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:34:35.693689 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.693676 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:34:35.693823 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.693806 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.693864 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.693840 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:34:35.694492 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.694460 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:34:35.694492 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.694483 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:34:35.694681 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.694488 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:34:35.694681 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.694553 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:34:35.694681 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.694509 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:34:35.694681 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.694577 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:34:35.695736 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.695719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.695834 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.695746 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:34:35.696407 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.696388 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:34:35.696502 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.696413 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:34:35.696502 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.696426 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:34:35.718038 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.718010 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-157.ec2.internal\" not found" node="ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.722131 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.722104 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-157.ec2.internal\" not found" node="ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.768603 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.768573 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:35.774325 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.774303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9f91dcc09ed8effbaa4a32384384a63-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal\" (UID: \"f9f91dcc09ed8effbaa4a32384384a63\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.774428 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.774333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49e9400fa5e85fa5702adbac4852ba2c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-157.ec2.internal\" (UID: \"49e9400fa5e85fa5702adbac4852ba2c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.774428 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.774353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9f91dcc09ed8effbaa4a32384384a63-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal\" (UID: \"f9f91dcc09ed8effbaa4a32384384a63\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.868959 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.868863 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:35.874627 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.874594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49e9400fa5e85fa5702adbac4852ba2c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-157.ec2.internal\" (UID: \"49e9400fa5e85fa5702adbac4852ba2c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.874732 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.874635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9f91dcc09ed8effbaa4a32384384a63-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal\" (UID: \"f9f91dcc09ed8effbaa4a32384384a63\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.874732 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.874661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9f91dcc09ed8effbaa4a32384384a63-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal\" (UID: \"f9f91dcc09ed8effbaa4a32384384a63\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.874732 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.874699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9f91dcc09ed8effbaa4a32384384a63-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal\" (UID: \"f9f91dcc09ed8effbaa4a32384384a63\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.874732 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.874702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9f91dcc09ed8effbaa4a32384384a63-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal\" (UID: \"f9f91dcc09ed8effbaa4a32384384a63\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.874732 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:35.874702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49e9400fa5e85fa5702adbac4852ba2c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-157.ec2.internal\" (UID: \"49e9400fa5e85fa5702adbac4852ba2c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal" Jul 28 17:34:35.976623 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:35.976573 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.024430 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.024397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal" Jul 28 17:34:36.026591 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.026570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" Jul 28 17:34:36.076732 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:36.076698 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.177011 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:36.176917 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.278091 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:36.278053 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.316159 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.316113 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 17:34:36.316677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.316323 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:34:36.378681 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:36.378644 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.400598 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.400552 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:29:36 +0000 UTC" deadline="2028-03-06 05:26:06.357862496 +0000 UTC" Jul 28 17:34:36.400598 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.400587 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14075h51m29.957279112s" Jul 28 17:34:36.479672 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:36.479629 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.488828 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.488791 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:34:36.564536 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:36.564491 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f91dcc09ed8effbaa4a32384384a63.slice/crio-1a697d01559f46b26feba922e4933634178781d6621c74d5507fcce2c27b4b43 WatchSource:0}: Error finding container 1a697d01559f46b26feba922e4933634178781d6621c74d5507fcce2c27b4b43: Status 404 returned error can't find the container with id 1a697d01559f46b26feba922e4933634178781d6621c74d5507fcce2c27b4b43 Jul 28 17:34:36.564821 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:36.564797 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e9400fa5e85fa5702adbac4852ba2c.slice/crio-d945a310cf6963750cb1fd8d9c41e3b0d75277cfc1986e611e694bf09672361f WatchSource:0}: Error finding container d945a310cf6963750cb1fd8d9c41e3b0d75277cfc1986e611e694bf09672361f: Status 404 returned error can't find the container with id d945a310cf6963750cb1fd8d9c41e3b0d75277cfc1986e611e694bf09672361f Jul 28 17:34:36.568849 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.568829 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:34:36.579867 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:36.579828 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.593869 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.593813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal" event={"ID":"49e9400fa5e85fa5702adbac4852ba2c","Type":"ContainerStarted","Data":"d945a310cf6963750cb1fd8d9c41e3b0d75277cfc1986e611e694bf09672361f"} Jul 28 17:34:36.594850 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.594823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" event={"ID":"f9f91dcc09ed8effbaa4a32384384a63","Type":"ContainerStarted","Data":"1a697d01559f46b26feba922e4933634178781d6621c74d5507fcce2c27b4b43"} Jul 28 17:34:36.679933 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:36.679896 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.780659 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:36.780581 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.880723 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:36.880683 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-157.ec2.internal\" not found" Jul 28 17:34:36.899466 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.899440 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:34:36.964846 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.964804 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:34:36.972112 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.972077 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" Jul 28 17:34:36.984640 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.984605 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:34:36.985912 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.985887 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal" Jul 28 17:34:36.995740 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:36.995716 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:34:37.354587 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.354548 2576 apiserver.go:52] "Watching apiserver" Jul 28 17:34:37.362466 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.362435 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 17:34:37.362859 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.362831 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-9jntc","openshift-multus/network-metrics-daemon-7hqdl","kube-system/global-pull-secret-syncer-zp4hx","kube-system/konnectivity-agent-l7q7r","openshift-multus/multus-additional-cni-plugins-tzk4n","openshift-network-diagnostics/network-check-target-mq7v6","openshift-network-operator/iptables-alerter-mxfwr","openshift-ovn-kubernetes/ovnkube-node-5jqv8","kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v","openshift-cluster-node-tuning-operator/tuned-v6ldd","openshift-dns/node-resolver-prxq5","openshift-image-registry/node-ca-67cb9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal"] Jul 28 17:34:37.364514 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.364488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.365577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.365552 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.366849 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.366826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.366933 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.366910 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:37.367547 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.367285 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 17:34:37.367547 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.367351 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5hpl6\"" Jul 28 17:34:37.367547 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.367531 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:34:37.367791 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.367622 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 17:34:37.368070 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.368049 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 17:34:37.368151 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.368135 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 17:34:37.368209 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.368171 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 17:34:37.368307 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.368288 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jdtlk\"" Jul 28 17:34:37.368387 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.368370 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:37.369480 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.369461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.371066 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.370896 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 17:34:37.371066 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.370915 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 17:34:37.371066 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.371027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:37.371297 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.371096 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:37.371297 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.371173 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xvgnl\"" Jul 28 17:34:37.371593 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.371551 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 17:34:37.371694 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.371593 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sn5x6\"" Jul 28 17:34:37.371948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.371882 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 17:34:37.371948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.371920 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 17:34:37.372128 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.371928 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 17:34:37.372360 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.372342 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 17:34:37.372852 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.372823 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 17:34:37.373055 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.373037 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.373890 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.373869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.375231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.375209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:37.375342 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.375318 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:37.377508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.377488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.379520 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.379532 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.379783 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.379859 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.380095 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4gtmw\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.380127 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.380396 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.380466 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.380675 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.380801 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fwsnf\"" Jul 28 17:34:37.381446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.381127 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 17:34:37.382078 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.381895 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 17:34:37.382816 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.382795 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.383367 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3bcbad70-e788-408a-8f1b-15c7bffafcfe-konnectivity-ca\") pod \"konnectivity-agent-l7q7r\" (UID: \"3bcbad70-e788-408a-8f1b-15c7bffafcfe\") " pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:37.383484 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-run-ovn\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.383484 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-run-ovn-kubernetes\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.383484 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c614aac7-e486-4623-a8d2-a011541fdeee-ovnkube-config\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.383657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-daemon-config\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.383657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-run-multus-certs\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.383657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmvr\" (UniqueName: \"kubernetes.io/projected/7c496d9d-d40c-4b43-920d-ceab761c6ad5-kube-api-access-qgmvr\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.383657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff31c586-d5f6-44de-bf64-b6539584c26d-kubelet-config\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.383657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff31c586-d5f6-44de-bf64-b6539584c26d-dbus\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.383657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c614aac7-e486-4623-a8d2-a011541fdeee-ovnkube-script-lib\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-run-openvswitch\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-run-netns\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-cni-netd\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-etc-kubernetes\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-run-k8s-cni-cncf-io\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7785bf5b-11ef-44ba-a530-f83e062952a9-serviceca\") pod \"node-ca-67cb9\" (UID: \"7785bf5b-11ef-44ba-a530-f83e062952a9\") " pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-var-lib-openvswitch\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-cni-dir\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c496d9d-d40c-4b43-920d-ceab761c6ad5-cni-binary-copy\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.383968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/141276f1-330f-4de4-a16e-8521cc402881-iptables-alerter-script\") pod \"iptables-alerter-mxfwr\" (UID: \"141276f1-330f-4de4-a16e-8521cc402881\") " pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.383973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ffba58f-7387-4f1f-b44c-a143ba089097-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp2x6\" (UniqueName: \"kubernetes.io/projected/8ffba58f-7387-4f1f-b44c-a143ba089097-kube-api-access-jp2x6\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2zs\" (UniqueName: \"kubernetes.io/projected/c614aac7-e486-4623-a8d2-a011541fdeee-kube-api-access-9c2zs\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-etc-openvswitch\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxp8c\" (UniqueName: \"kubernetes.io/projected/7785bf5b-11ef-44ba-a530-f83e062952a9-kube-api-access-dxp8c\") pod \"node-ca-67cb9\" (UID: \"7785bf5b-11ef-44ba-a530-f83e062952a9\") " pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ffba58f-7387-4f1f-b44c-a143ba089097-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-kubelet\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-log-socket\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-cni-bin\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-run-netns\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdq9\" (UniqueName: \"kubernetes.io/projected/141276f1-330f-4de4-a16e-8521cc402881-kube-api-access-vzdq9\") pod \"iptables-alerter-mxfwr\" (UID: \"141276f1-330f-4de4-a16e-8521cc402881\") " pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-system-cni-dir\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-os-release\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-socket-dir-parent\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.384523 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-var-lib-cni-multus\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-conf-dir\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-systemd-units\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-var-lib-cni-bin\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-hostroot\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7785bf5b-11ef-44ba-a530-f83e062952a9-host\") pod \"node-ca-67cb9\" (UID: \"7785bf5b-11ef-44ba-a530-f83e062952a9\") " pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3bcbad70-e788-408a-8f1b-15c7bffafcfe-agent-certs\") pod \"konnectivity-agent-l7q7r\" (UID: \"3bcbad70-e788-408a-8f1b-15c7bffafcfe\") " pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c614aac7-e486-4623-a8d2-a011541fdeee-env-overrides\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-system-cni-dir\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-var-lib-kubelet\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/141276f1-330f-4de4-a16e-8521cc402881-host-slash\") pod \"iptables-alerter-mxfwr\" (UID: \"141276f1-330f-4de4-a16e-8521cc402881\") " pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-cnibin\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-os-release\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ffba58f-7387-4f1f-b44c-a143ba089097-cni-binary-copy\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.385333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-node-log\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.386138 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c614aac7-e486-4623-a8d2-a011541fdeee-ovn-node-metrics-cert\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.386138 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-slash\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.386138 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-run-systemd\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.386138 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.384959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-cnibin\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.386138 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.385145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.386138 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.385933 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nzlx8\"" Jul 28 17:34:37.386582 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.386481 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 17:34:37.386863 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.386810 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qcbp\"" Jul 28 17:34:37.387708 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.387686 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:34:37.387804 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.387764 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 17:34:37.387804 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.387770 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-88zs2\"" Jul 28 17:34:37.387920 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.387896 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 17:34:37.391202 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.391178 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:34:37.419979 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.419954 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-482d9" Jul 28 17:34:37.429700 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.429636 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-482d9" Jul 28 17:34:37.473574 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.473538 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 17:34:37.485537 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-systemd-units\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-var-lib-cni-bin\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-hostroot\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7785bf5b-11ef-44ba-a530-f83e062952a9-host\") pod \"node-ca-67cb9\" (UID: \"7785bf5b-11ef-44ba-a530-f83e062952a9\") " pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-systemd-units\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-tuned\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485648 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-var-lib-cni-bin\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-hostroot\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7785bf5b-11ef-44ba-a530-f83e062952a9-host\") pod \"node-ca-67cb9\" (UID: \"7785bf5b-11ef-44ba-a530-f83e062952a9\") " pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.485743 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3bcbad70-e788-408a-8f1b-15c7bffafcfe-agent-certs\") pod \"konnectivity-agent-l7q7r\" (UID: \"3bcbad70-e788-408a-8f1b-15c7bffafcfe\") " pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c614aac7-e486-4623-a8d2-a011541fdeee-env-overrides\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-system-cni-dir\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-modprobe-d\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-system-cni-dir\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqk4n\" (UniqueName: \"kubernetes.io/projected/c6998d3c-ff71-41f2-972c-8f5bfcee0614-kube-api-access-xqk4n\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-var-lib-kubelet\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.485994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/141276f1-330f-4de4-a16e-8521cc402881-host-slash\") pod \"iptables-alerter-mxfwr\" (UID: \"141276f1-330f-4de4-a16e-8521cc402881\") " pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-cnibin\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-etc-selinux\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-os-release\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/141276f1-330f-4de4-a16e-8521cc402881-host-slash\") pod \"iptables-alerter-mxfwr\" (UID: \"141276f1-330f-4de4-a16e-8521cc402881\") " pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ffba58f-7387-4f1f-b44c-a143ba089097-cni-binary-copy\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486125 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-os-release\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ffba58f-7387-4f1f-b44c-a143ba089097-cnibin\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.486306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-var-lib-kubelet\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c614aac7-e486-4623-a8d2-a011541fdeee-env-overrides\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-node-log\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c614aac7-e486-4623-a8d2-a011541fdeee-ovn-node-metrics-cert\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-slash\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-run-systemd\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-cnibin\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtck9\" (UniqueName: \"kubernetes.io/projected/de3de89b-e0c9-4347-b778-e7893bef0f5e-kube-api-access-mtck9\") pod \"node-resolver-prxq5\" (UID: \"de3de89b-e0c9-4347-b778-e7893bef0f5e\") " pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486466 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-node-log\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3bcbad70-e788-408a-8f1b-15c7bffafcfe-konnectivity-ca\") pod \"konnectivity-agent-l7q7r\" (UID: \"3bcbad70-e788-408a-8f1b-15c7bffafcfe\") " pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-run-systemd\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-run-ovn\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-run-ovn-kubernetes\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c614aac7-e486-4623-a8d2-a011541fdeee-ovnkube-config\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-daemon-config\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-cnibin\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ffba58f-7387-4f1f-b44c-a143ba089097-cni-binary-copy\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51112e6a-064a-4a92-ad7f-bd511b166d9f-tmp\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.487046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-slash\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-run-multus-certs\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmvr\" (UniqueName: \"kubernetes.io/projected/7c496d9d-d40c-4b43-920d-ceab761c6ad5-kube-api-access-qgmvr\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff31c586-d5f6-44de-bf64-b6539584c26d-kubelet-config\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff31c586-d5f6-44de-bf64-b6539584c26d-dbus\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.486979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-var-lib-kubelet\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-host\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c614aac7-e486-4623-a8d2-a011541fdeee-ovnkube-script-lib\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-socket-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-run-openvswitch\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3bcbad70-e788-408a-8f1b-15c7bffafcfe-konnectivity-ca\") pod \"konnectivity-agent-l7q7r\" (UID: \"3bcbad70-e788-408a-8f1b-15c7bffafcfe\") " pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-run-netns\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-lib-modules\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-run-ovn\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-cni-netd\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.487900 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-etc-kubernetes\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487351 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-run-ovn-kubernetes\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-sysconfig\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-run-multus-certs\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de3de89b-e0c9-4347-b778-e7893bef0f5e-hosts-file\") pod \"node-resolver-prxq5\" (UID: \"de3de89b-e0c9-4347-b778-e7893bef0f5e\") " pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd9g9\" (UniqueName: \"kubernetes.io/projected/0080a572-035a-4696-b249-18c75ec5e83c-kube-api-access-fd9g9\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-run-k8s-cni-cncf-io\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487476 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-etc-kubernetes\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7785bf5b-11ef-44ba-a530-f83e062952a9-serviceca\") pod \"node-ca-67cb9\" (UID: \"7785bf5b-11ef-44ba-a530-f83e062952a9\") " pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-kubernetes\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-run-openvswitch\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487619 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-run-netns\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-cni-netd\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487676 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-run-k8s-cni-cncf-io\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff31c586-d5f6-44de-bf64-b6539584c26d-kubelet-config\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-daemon-config\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c614aac7-e486-4623-a8d2-a011541fdeee-ovnkube-config\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff31c586-d5f6-44de-bf64-b6539584c26d-dbus\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.488771 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.487930 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-run\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.487981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3de89b-e0c9-4347-b778-e7893bef0f5e-tmp-dir\") pod \"node-resolver-prxq5\" (UID: \"de3de89b-e0c9-4347-b778-e7893bef0f5e\") " pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.488006 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret podName:ff31c586-d5f6-44de-bf64-b6539584c26d nodeName:}" failed. No retries permitted until 2026-07-28 17:34:37.987982263 +0000 UTC m=+2.960489808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret") pod "global-pull-secret-syncer-zp4hx" (UID: "ff31c586-d5f6-44de-bf64-b6539584c26d") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-var-lib-openvswitch\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7785bf5b-11ef-44ba-a530-f83e062952a9-serviceca\") pod \"node-ca-67cb9\" (UID: \"7785bf5b-11ef-44ba-a530-f83e062952a9\") " pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-cni-dir\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c496d9d-d40c-4b43-920d-ceab761c6ad5-cni-binary-copy\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/141276f1-330f-4de4-a16e-8521cc402881-iptables-alerter-script\") pod \"iptables-alerter-mxfwr\" (UID: \"141276f1-330f-4de4-a16e-8521cc402881\") " pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-var-lib-openvswitch\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ffba58f-7387-4f1f-b44c-a143ba089097-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp2x6\" (UniqueName: \"kubernetes.io/projected/8ffba58f-7387-4f1f-b44c-a143ba089097-kube-api-access-jp2x6\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488229 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-cni-dir\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-systemd\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2zs\" (UniqueName: \"kubernetes.io/projected/c614aac7-e486-4623-a8d2-a011541fdeee-kube-api-access-9c2zs\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.489677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c614aac7-e486-4623-a8d2-a011541fdeee-ovnkube-script-lib\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-etc-openvswitch\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxp8c\" (UniqueName: \"kubernetes.io/projected/7785bf5b-11ef-44ba-a530-f83e062952a9-kube-api-access-dxp8c\") pod \"node-ca-67cb9\" (UID: \"7785bf5b-11ef-44ba-a530-f83e062952a9\") " pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ffba58f-7387-4f1f-b44c-a143ba089097-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-sysctl-d\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xkz\" (UniqueName: \"kubernetes.io/projected/51112e6a-064a-4a92-ad7f-bd511b166d9f-kube-api-access-b6xkz\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-kubelet\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-log-socket\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-cni-bin\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-run-netns\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdq9\" (UniqueName: \"kubernetes.io/projected/141276f1-330f-4de4-a16e-8521cc402881-kube-api-access-vzdq9\") pod \"iptables-alerter-mxfwr\" (UID: \"141276f1-330f-4de4-a16e-8521cc402881\") " pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-registration-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-system-cni-dir\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-os-release\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-socket-dir-parent\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-var-lib-cni-multus\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.490580 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-conf-dir\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ffba58f-7387-4f1f-b44c-a143ba089097-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-device-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-sysctl-conf\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-sys\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.488881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-sys-fs\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-kubelet\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ffba58f-7387-4f1f-b44c-a143ba089097-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c496d9d-d40c-4b43-920d-ceab761c6ad5-cni-binary-copy\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-log-socket\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/141276f1-330f-4de4-a16e-8521cc402881-iptables-alerter-script\") pod \"iptables-alerter-mxfwr\" (UID: \"141276f1-330f-4de4-a16e-8521cc402881\") " pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489373 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-run-netns\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-conf-dir\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-os-release\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-multus-socket-dir-parent\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-etc-openvswitch\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.491508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489516 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-host-var-lib-cni-multus\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.492218 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c496d9d-d40c-4b43-920d-ceab761c6ad5-system-cni-dir\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.492218 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.492218 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.489578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c614aac7-e486-4623-a8d2-a011541fdeee-host-cni-bin\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.492218 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.490603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c614aac7-e486-4623-a8d2-a011541fdeee-ovn-node-metrics-cert\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.492218 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.490825 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3bcbad70-e788-408a-8f1b-15c7bffafcfe-agent-certs\") pod \"konnectivity-agent-l7q7r\" (UID: \"3bcbad70-e788-408a-8f1b-15c7bffafcfe\") " pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:37.500042 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.499967 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:34:37.500042 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.499998 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:34:37.500042 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.500039 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lhrln for pod openshift-network-diagnostics/network-check-target-mq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:37.500574 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.500115 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln podName:e7d31075-9a8d-4306-9f59-c82ae381c980 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:38.000089588 +0000 UTC m=+2.972597125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lhrln" (UniqueName: "kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln") pod "network-check-target-mq7v6" (UID: "e7d31075-9a8d-4306-9f59-c82ae381c980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:37.501960 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.501926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmvr\" (UniqueName: \"kubernetes.io/projected/7c496d9d-d40c-4b43-920d-ceab761c6ad5-kube-api-access-qgmvr\") pod \"multus-9jntc\" (UID: \"7c496d9d-d40c-4b43-920d-ceab761c6ad5\") " pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.503811 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.503783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2zs\" (UniqueName: \"kubernetes.io/projected/c614aac7-e486-4623-a8d2-a011541fdeee-kube-api-access-9c2zs\") pod \"ovnkube-node-5jqv8\" (UID: \"c614aac7-e486-4623-a8d2-a011541fdeee\") " pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.504447 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.504412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxp8c\" (UniqueName: \"kubernetes.io/projected/7785bf5b-11ef-44ba-a530-f83e062952a9-kube-api-access-dxp8c\") pod \"node-ca-67cb9\" (UID: \"7785bf5b-11ef-44ba-a530-f83e062952a9\") " pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.506238 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.506161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdq9\" (UniqueName: \"kubernetes.io/projected/141276f1-330f-4de4-a16e-8521cc402881-kube-api-access-vzdq9\") pod \"iptables-alerter-mxfwr\" (UID: \"141276f1-330f-4de4-a16e-8521cc402881\") " pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.506377 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.506351 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp2x6\" (UniqueName: \"kubernetes.io/projected/8ffba58f-7387-4f1f-b44c-a143ba089097-kube-api-access-jp2x6\") pod \"multus-additional-cni-plugins-tzk4n\" (UID: \"8ffba58f-7387-4f1f-b44c-a143ba089097\") " pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.589256 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-sysconfig\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589256 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de3de89b-e0c9-4347-b778-e7893bef0f5e-hosts-file\") pod \"node-resolver-prxq5\" (UID: \"de3de89b-e0c9-4347-b778-e7893bef0f5e\") " pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd9g9\" (UniqueName: \"kubernetes.io/projected/0080a572-035a-4696-b249-18c75ec5e83c-kube-api-access-fd9g9\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-kubernetes\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-run\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3de89b-e0c9-4347-b778-e7893bef0f5e-tmp-dir\") pod \"node-resolver-prxq5\" (UID: \"de3de89b-e0c9-4347-b778-e7893bef0f5e\") " pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589366 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-sysconfig\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589378 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de3de89b-e0c9-4347-b778-e7893bef0f5e-hosts-file\") pod \"node-resolver-prxq5\" (UID: \"de3de89b-e0c9-4347-b778-e7893bef0f5e\") " pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-systemd\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-kubernetes\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-run\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-systemd\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-sysctl-d\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xkz\" (UniqueName: \"kubernetes.io/projected/51112e6a-064a-4a92-ad7f-bd511b166d9f-kube-api-access-b6xkz\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-registration-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589599 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-sysctl-d\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.589623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-device-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-sysctl-conf\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-sys\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-sys-fs\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-device-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-tuned\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589763 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-sys\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-registration-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3de89b-e0c9-4347-b778-e7893bef0f5e-tmp-dir\") pod \"node-resolver-prxq5\" (UID: \"de3de89b-e0c9-4347-b778-e7893bef0f5e\") " pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589855 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-sys-fs\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-modprobe-d\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-modprobe-d\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589902 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-sysctl-conf\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqk4n\" (UniqueName: \"kubernetes.io/projected/c6998d3c-ff71-41f2-972c-8f5bfcee0614-kube-api-access-xqk4n\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-etc-selinux\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.589977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtck9\" (UniqueName: \"kubernetes.io/projected/de3de89b-e0c9-4347-b778-e7893bef0f5e-kube-api-access-mtck9\") pod \"node-resolver-prxq5\" (UID: \"de3de89b-e0c9-4347-b778-e7893bef0f5e\") " pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51112e6a-064a-4a92-ad7f-bd511b166d9f-tmp\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.590577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-var-lib-kubelet\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590065 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-host\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-socket-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-var-lib-kubelet\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-lib-modules\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-host\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590117 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-etc-selinux\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.590301 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0080a572-035a-4696-b249-18c75ec5e83c-socket-dir\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.590334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51112e6a-064a-4a92-ad7f-bd511b166d9f-lib-modules\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.591472 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.590361 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs podName:c6998d3c-ff71-41f2-972c-8f5bfcee0614 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:38.090341748 +0000 UTC m=+3.062849300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs") pod "network-metrics-daemon-7hqdl" (UID: "c6998d3c-ff71-41f2-972c-8f5bfcee0614") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:37.592659 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.592624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51112e6a-064a-4a92-ad7f-bd511b166d9f-etc-tuned\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.593047 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.593008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51112e6a-064a-4a92-ad7f-bd511b166d9f-tmp\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.603464 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.603400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd9g9\" (UniqueName: \"kubernetes.io/projected/0080a572-035a-4696-b249-18c75ec5e83c-kube-api-access-fd9g9\") pod \"aws-ebs-csi-driver-node-92n7v\" (UID: \"0080a572-035a-4696-b249-18c75ec5e83c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.604524 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.604498 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtck9\" (UniqueName: \"kubernetes.io/projected/de3de89b-e0c9-4347-b778-e7893bef0f5e-kube-api-access-mtck9\") pod \"node-resolver-prxq5\" (UID: \"de3de89b-e0c9-4347-b778-e7893bef0f5e\") " pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.605109 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.605046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xkz\" (UniqueName: \"kubernetes.io/projected/51112e6a-064a-4a92-ad7f-bd511b166d9f-kube-api-access-b6xkz\") pod \"tuned-v6ldd\" (UID: \"51112e6a-064a-4a92-ad7f-bd511b166d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.605109 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.605046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqk4n\" (UniqueName: \"kubernetes.io/projected/c6998d3c-ff71-41f2-972c-8f5bfcee0614-kube-api-access-xqk4n\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:37.683033 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.682995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mxfwr" Jul 28 17:34:37.689988 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.689953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-67cb9" Jul 28 17:34:37.700220 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.700192 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:37.705961 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.705935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" Jul 28 17:34:37.713820 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.713797 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:34:37.721892 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.721853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9jntc" Jul 28 17:34:37.728919 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.728875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" Jul 28 17:34:37.737082 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.737040 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" Jul 28 17:34:37.749274 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.749237 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-prxq5" Jul 28 17:34:37.879236 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.879134 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:34:37.992999 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:37.992957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:37.993168 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.993147 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:37.993261 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:37.993248 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret podName:ff31c586-d5f6-44de-bf64-b6539584c26d nodeName:}" failed. No retries permitted until 2026-07-28 17:34:38.993225966 +0000 UTC m=+3.965733503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret") pod "global-pull-secret-syncer-zp4hx" (UID: "ff31c586-d5f6-44de-bf64-b6539584c26d") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:38.094188 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.094144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:38.094395 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.094216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:38.094395 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:38.094306 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:38.094395 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:38.094350 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:34:38.094395 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:38.094375 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:34:38.094395 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:38.094390 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lhrln for pod openshift-network-diagnostics/network-check-target-mq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:38.094395 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:38.094394 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs podName:c6998d3c-ff71-41f2-972c-8f5bfcee0614 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:39.094371446 +0000 UTC m=+4.066878999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs") pod "network-metrics-daemon-7hqdl" (UID: "c6998d3c-ff71-41f2-972c-8f5bfcee0614") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:38.094737 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:38.094444 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln podName:e7d31075-9a8d-4306-9f59-c82ae381c980 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:39.094426689 +0000 UTC m=+4.066934220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhrln" (UniqueName: "kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln") pod "network-check-target-mq7v6" (UID: "e7d31075-9a8d-4306-9f59-c82ae381c980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:38.432687 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.432647 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:29:38 +0000 UTC" deadline="2028-02-14 17:17:32.392306392 +0000 UTC" Jul 28 17:34:38.432687 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.432680 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13583h42m53.959629682s" Jul 28 17:34:38.488190 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:38.488153 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0080a572_035a_4696_b249_18c75ec5e83c.slice/crio-f34f8261d7017d093d7d706bc39fb664747c1f2aa82a022e83a0349e45c020ee WatchSource:0}: Error finding container f34f8261d7017d093d7d706bc39fb664747c1f2aa82a022e83a0349e45c020ee: Status 404 returned error can't find the container with id f34f8261d7017d093d7d706bc39fb664747c1f2aa82a022e83a0349e45c020ee Jul 28 17:34:38.489509 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:38.489486 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7785bf5b_11ef_44ba_a530_f83e062952a9.slice/crio-9097a036129c5eb10562383a7a82fc1dab4dbdd138c4d1fa807ebf86afb22705 WatchSource:0}: Error finding container 9097a036129c5eb10562383a7a82fc1dab4dbdd138c4d1fa807ebf86afb22705: Status 404 returned error can't find the container with id 9097a036129c5eb10562383a7a82fc1dab4dbdd138c4d1fa807ebf86afb22705 Jul 28 17:34:38.489807 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:38.489726 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c496d9d_d40c_4b43_920d_ceab761c6ad5.slice/crio-9296e7e89deb756c335dd520bcaa355dc88d7ff37a97c6a33db4d8cd7f8e11c8 WatchSource:0}: Error finding container 9296e7e89deb756c335dd520bcaa355dc88d7ff37a97c6a33db4d8cd7f8e11c8: Status 404 returned error can't find the container with id 9296e7e89deb756c335dd520bcaa355dc88d7ff37a97c6a33db4d8cd7f8e11c8 Jul 28 17:34:38.491004 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:38.490965 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcbad70_e788_408a_8f1b_15c7bffafcfe.slice/crio-f4e015ae0a1b88805a46d1179452354c2544e00d9d8303636ae82dd1692e8337 WatchSource:0}: Error finding container f4e015ae0a1b88805a46d1179452354c2544e00d9d8303636ae82dd1692e8337: Status 404 returned error can't find the container with id f4e015ae0a1b88805a46d1179452354c2544e00d9d8303636ae82dd1692e8337 Jul 28 17:34:38.493168 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:38.493133 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc614aac7_e486_4623_a8d2_a011541fdeee.slice/crio-6cdf72b9ac6d2712e23d09b6d449764ed052ff33daa4912b9fe417a24b50dbdd WatchSource:0}: Error finding container 6cdf72b9ac6d2712e23d09b6d449764ed052ff33daa4912b9fe417a24b50dbdd: Status 404 returned error can't find the container with id 6cdf72b9ac6d2712e23d09b6d449764ed052ff33daa4912b9fe417a24b50dbdd Jul 28 17:34:38.494648 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:38.493853 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3de89b_e0c9_4347_b778_e7893bef0f5e.slice/crio-3d17fa7aa7d573b52989415535b88d452edadb0646293f36d64c61889148ef7e WatchSource:0}: Error finding container 3d17fa7aa7d573b52989415535b88d452edadb0646293f36d64c61889148ef7e: Status 404 returned error can't find the container with id 3d17fa7aa7d573b52989415535b88d452edadb0646293f36d64c61889148ef7e Jul 28 17:34:38.494648 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:38.494506 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141276f1_330f_4de4_a16e_8521cc402881.slice/crio-faae4360fa99aab86caf9fc0605403500ce78f3a8b276f672ce91f266fca2048 WatchSource:0}: Error finding container faae4360fa99aab86caf9fc0605403500ce78f3a8b276f672ce91f266fca2048: Status 404 returned error can't find the container with id faae4360fa99aab86caf9fc0605403500ce78f3a8b276f672ce91f266fca2048 Jul 28 17:34:38.496406 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:34:38.496343 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffba58f_7387_4f1f_b44c_a143ba089097.slice/crio-6ce53e9940a1c803258d5d723047742673347524ddcea0ea031182e9ac868c35 WatchSource:0}: Error finding container 6ce53e9940a1c803258d5d723047742673347524ddcea0ea031182e9ac868c35: Status 404 returned error can't find the container with id 6ce53e9940a1c803258d5d723047742673347524ddcea0ea031182e9ac868c35 Jul 28 17:34:38.590015 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.589990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:38.590125 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:38.590101 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:38.600703 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.600676 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal" event={"ID":"49e9400fa5e85fa5702adbac4852ba2c","Type":"ContainerStarted","Data":"82fce787abe626d62acf5cee56d1d5089a15c577501704a042873cde44a2b6e4"} Jul 28 17:34:38.602026 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.602003 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7q7r" event={"ID":"3bcbad70-e788-408a-8f1b-15c7bffafcfe","Type":"ContainerStarted","Data":"f4e015ae0a1b88805a46d1179452354c2544e00d9d8303636ae82dd1692e8337"} Jul 28 17:34:38.603331 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.603305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9jntc" event={"ID":"7c496d9d-d40c-4b43-920d-ceab761c6ad5","Type":"ContainerStarted","Data":"9296e7e89deb756c335dd520bcaa355dc88d7ff37a97c6a33db4d8cd7f8e11c8"} Jul 28 17:34:38.604790 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.604746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" event={"ID":"51112e6a-064a-4a92-ad7f-bd511b166d9f","Type":"ContainerStarted","Data":"0b33584771027ad650428bc6156f2d8d75e89295c7194cf70124e2d987ef8a6a"} Jul 28 17:34:38.605946 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.605926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" event={"ID":"8ffba58f-7387-4f1f-b44c-a143ba089097","Type":"ContainerStarted","Data":"6ce53e9940a1c803258d5d723047742673347524ddcea0ea031182e9ac868c35"} Jul 28 17:34:38.607028 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.607007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mxfwr" event={"ID":"141276f1-330f-4de4-a16e-8521cc402881","Type":"ContainerStarted","Data":"faae4360fa99aab86caf9fc0605403500ce78f3a8b276f672ce91f266fca2048"} Jul 28 17:34:38.608124 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.608104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-prxq5" event={"ID":"de3de89b-e0c9-4347-b778-e7893bef0f5e","Type":"ContainerStarted","Data":"3d17fa7aa7d573b52989415535b88d452edadb0646293f36d64c61889148ef7e"} Jul 28 17:34:38.609189 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.609165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" event={"ID":"c614aac7-e486-4623-a8d2-a011541fdeee","Type":"ContainerStarted","Data":"6cdf72b9ac6d2712e23d09b6d449764ed052ff33daa4912b9fe417a24b50dbdd"} Jul 28 17:34:38.610502 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.610480 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-67cb9" event={"ID":"7785bf5b-11ef-44ba-a530-f83e062952a9","Type":"ContainerStarted","Data":"9097a036129c5eb10562383a7a82fc1dab4dbdd138c4d1fa807ebf86afb22705"} Jul 28 17:34:38.611601 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.611581 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" event={"ID":"0080a572-035a-4696-b249-18c75ec5e83c","Type":"ContainerStarted","Data":"f34f8261d7017d093d7d706bc39fb664747c1f2aa82a022e83a0349e45c020ee"} Jul 28 17:34:38.615319 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:38.615280 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-157.ec2.internal" podStartSLOduration=1.6152678919999999 podStartE2EDuration="1.615267892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:38.61478166 +0000 UTC m=+3.587289215" watchObservedRunningTime="2026-07-28 17:34:38.615267892 +0000 UTC m=+3.587775439" Jul 28 17:34:39.002399 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:39.001677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:39.002399 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.001965 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:39.002399 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.002062 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret podName:ff31c586-d5f6-44de-bf64-b6539584c26d nodeName:}" failed. No retries permitted until 2026-07-28 17:34:41.002034237 +0000 UTC m=+5.974541772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret") pod "global-pull-secret-syncer-zp4hx" (UID: "ff31c586-d5f6-44de-bf64-b6539584c26d") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:39.102439 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:39.102394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:39.102616 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:39.102494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:39.102681 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.102634 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:39.102734 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.102700 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs podName:c6998d3c-ff71-41f2-972c-8f5bfcee0614 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:41.102680659 +0000 UTC m=+6.075188207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs") pod "network-metrics-daemon-7hqdl" (UID: "c6998d3c-ff71-41f2-972c-8f5bfcee0614") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:39.103154 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.103132 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:34:39.103278 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.103160 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:34:39.103278 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.103174 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lhrln for pod openshift-network-diagnostics/network-check-target-mq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:39.103278 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.103222 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln podName:e7d31075-9a8d-4306-9f59-c82ae381c980 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:41.10320464 +0000 UTC m=+6.075712177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhrln" (UniqueName: "kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln") pod "network-check-target-mq7v6" (UID: "e7d31075-9a8d-4306-9f59-c82ae381c980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:39.434290 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:39.434114 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:29:38 +0000 UTC" deadline="2028-03-02 03:57:22.584797304 +0000 UTC" Jul 28 17:34:39.434290 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:39.434197 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13978h22m43.150605543s" Jul 28 17:34:39.591439 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:39.591406 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:39.591602 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.591565 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:39.591667 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:39.591631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:39.591792 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:39.591767 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:39.621124 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:39.621062 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9f91dcc09ed8effbaa4a32384384a63" containerID="3dbfb7d418f00f01e2a5d28566d19935a36ab2e7817c364297bbf6e2e9828837" exitCode=0 Jul 28 17:34:39.622426 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:39.622202 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" event={"ID":"f9f91dcc09ed8effbaa4a32384384a63","Type":"ContainerDied","Data":"3dbfb7d418f00f01e2a5d28566d19935a36ab2e7817c364297bbf6e2e9828837"} Jul 28 17:34:40.590333 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:40.590291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:40.590848 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:40.590455 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:40.644052 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:40.643965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" event={"ID":"f9f91dcc09ed8effbaa4a32384384a63","Type":"ContainerStarted","Data":"5df1dfd25616e800e8dce776f5c591998d87f2995eb474a0ce20824c16ce0910"} Jul 28 17:34:41.016447 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:41.016352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:41.016631 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.016607 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:41.016707 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.016693 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret podName:ff31c586-d5f6-44de-bf64-b6539584c26d nodeName:}" failed. No retries permitted until 2026-07-28 17:34:45.016671019 +0000 UTC m=+9.989178551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret") pod "global-pull-secret-syncer-zp4hx" (UID: "ff31c586-d5f6-44de-bf64-b6539584c26d") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:41.117600 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:41.117560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:41.117801 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:41.117663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:41.117864 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.117806 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:41.117919 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.117881 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs podName:c6998d3c-ff71-41f2-972c-8f5bfcee0614 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:45.117853723 +0000 UTC m=+10.090361260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs") pod "network-metrics-daemon-7hqdl" (UID: "c6998d3c-ff71-41f2-972c-8f5bfcee0614") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:41.118423 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.118401 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:34:41.118509 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.118433 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:34:41.118509 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.118447 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lhrln for pod openshift-network-diagnostics/network-check-target-mq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:41.118509 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.118497 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln podName:e7d31075-9a8d-4306-9f59-c82ae381c980 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:45.118480936 +0000 UTC m=+10.090988480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhrln" (UniqueName: "kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln") pod "network-check-target-mq7v6" (UID: "e7d31075-9a8d-4306-9f59-c82ae381c980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:41.593555 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:41.592728 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:41.593555 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.592880 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:41.593555 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:41.593363 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:41.593555 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:41.593463 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:42.590668 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:42.590632 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:42.590879 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:42.590793 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:43.591457 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:43.590888 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:43.591457 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:43.591035 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:43.591457 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:43.591452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:43.592071 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:43.591572 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:44.590123 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:44.590082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:44.590345 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:44.590258 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:45.052815 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:45.052771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:45.053343 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.052937 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:45.053343 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.053023 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret podName:ff31c586-d5f6-44de-bf64-b6539584c26d nodeName:}" failed. No retries permitted until 2026-07-28 17:34:53.053000775 +0000 UTC m=+18.025508315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret") pod "global-pull-secret-syncer-zp4hx" (UID: "ff31c586-d5f6-44de-bf64-b6539584c26d") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:45.153981 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:45.153721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:45.153981 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:45.153824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:45.153981 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.153962 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:45.154337 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.154005 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:34:45.154337 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.154035 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:34:45.154337 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.154039 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs podName:c6998d3c-ff71-41f2-972c-8f5bfcee0614 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:53.154017738 +0000 UTC m=+18.126525276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs") pod "network-metrics-daemon-7hqdl" (UID: "c6998d3c-ff71-41f2-972c-8f5bfcee0614") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:45.154337 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.154050 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lhrln for pod openshift-network-diagnostics/network-check-target-mq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:45.154337 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.154105 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln podName:e7d31075-9a8d-4306-9f59-c82ae381c980 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:53.154087558 +0000 UTC m=+18.126595104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhrln" (UniqueName: "kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln") pod "network-check-target-mq7v6" (UID: "e7d31075-9a8d-4306-9f59-c82ae381c980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:45.591207 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:45.591137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:45.591383 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.591247 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:45.591383 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:45.591318 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:45.591525 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:45.591389 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:46.589972 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:46.589930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:46.590438 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:46.590087 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:47.590767 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:47.590288 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:47.590767 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:47.590415 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:47.591314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:47.590830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:47.591314 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:47.590942 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:48.590015 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:48.589977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:48.590194 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:48.590135 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:49.590416 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:49.590369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:49.590416 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:49.590418 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:49.590997 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:49.590504 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:49.590997 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:49.590609 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:50.589804 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:50.589761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:50.589969 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:50.589911 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:51.593488 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:51.593462 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:51.593904 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:51.593462 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:51.593904 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:51.593559 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:51.593904 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:51.593646 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:52.589727 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:52.589675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:52.589964 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:52.589813 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:53.113864 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:53.113834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:53.114279 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.113969 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:53.114279 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.114021 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret podName:ff31c586-d5f6-44de-bf64-b6539584c26d nodeName:}" failed. No retries permitted until 2026-07-28 17:35:09.114007295 +0000 UTC m=+34.086514825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret") pod "global-pull-secret-syncer-zp4hx" (UID: "ff31c586-d5f6-44de-bf64-b6539584c26d") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:34:53.214442 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:53.214403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:53.214636 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:53.214474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:53.214636 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.214536 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:53.214636 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.214603 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:34:53.214636 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.214617 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:34:53.214636 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.214629 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lhrln for pod openshift-network-diagnostics/network-check-target-mq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:53.214948 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.214618 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs podName:c6998d3c-ff71-41f2-972c-8f5bfcee0614 nodeName:}" failed. No retries permitted until 2026-07-28 17:35:09.214597972 +0000 UTC m=+34.187105504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs") pod "network-metrics-daemon-7hqdl" (UID: "c6998d3c-ff71-41f2-972c-8f5bfcee0614") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:34:53.214948 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.214697 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln podName:e7d31075-9a8d-4306-9f59-c82ae381c980 nodeName:}" failed. No retries permitted until 2026-07-28 17:35:09.214680355 +0000 UTC m=+34.187187900 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhrln" (UniqueName: "kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln") pod "network-check-target-mq7v6" (UID: "e7d31075-9a8d-4306-9f59-c82ae381c980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:34:53.590583 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:53.590540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:53.590583 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:53.590588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:53.590778 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.590671 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:53.590814 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:53.590784 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:54.590383 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:54.590326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:54.590790 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:54.590466 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:55.591409 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:55.591365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:55.591868 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:55.591469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:55.591868 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:55.591481 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:55.591868 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:55.591584 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:56.589953 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:56.589909 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:56.590145 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:56.590045 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:57.590049 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:57.590014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:57.590628 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:57.590026 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:57.590628 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:57.590160 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:57.590628 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:57.590249 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:58.590405 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.589976 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:34:58.591195 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:58.590541 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:34:58.686255 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.686224 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ffba58f-7387-4f1f-b44c-a143ba089097" containerID="a1a45fd86cfebea5a680611fd8cb6a9393e50c1c35205a1fd1c4ac2fb931b4d3" exitCode=0 Jul 28 17:34:58.686440 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.686301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" event={"ID":"8ffba58f-7387-4f1f-b44c-a143ba089097","Type":"ContainerDied","Data":"a1a45fd86cfebea5a680611fd8cb6a9393e50c1c35205a1fd1c4ac2fb931b4d3"} Jul 28 17:34:58.687885 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.687862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-prxq5" event={"ID":"de3de89b-e0c9-4347-b778-e7893bef0f5e","Type":"ContainerStarted","Data":"8099584063f523a9380d24b89bd748996731da70afb071158af6cdd0f1afc425"} Jul 28 17:34:58.691497 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.691473 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" event={"ID":"c614aac7-e486-4623-a8d2-a011541fdeee","Type":"ContainerStarted","Data":"a58ed88c106dacb12e57586ae89fcac86cc35c3601b70ef9e7ef5f4096e462ae"} Jul 28 17:34:58.691602 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.691549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" event={"ID":"c614aac7-e486-4623-a8d2-a011541fdeee","Type":"ContainerStarted","Data":"56b612ad66d4551b794906c159907223ee646f999e13cbd8f829229e05143e5c"} Jul 28 17:34:58.691662 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.691601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" event={"ID":"c614aac7-e486-4623-a8d2-a011541fdeee","Type":"ContainerStarted","Data":"2a3639da8c7b76adaab4d0f69f2561272883d90d61af2eeea30007db3241b4c0"} Jul 28 17:34:58.691662 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.691650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" event={"ID":"c614aac7-e486-4623-a8d2-a011541fdeee","Type":"ContainerStarted","Data":"fe00453a3c299ac00cefe1098b525ed6c41c093411374de463c391748eda394c"} Jul 28 17:34:58.691764 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.691697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" event={"ID":"c614aac7-e486-4623-a8d2-a011541fdeee","Type":"ContainerStarted","Data":"2c6a8f9fc354f22638d7cc4ad883ffe5ada47c0b259531a0e051bb70bec61044"} Jul 28 17:34:58.691764 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.691742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" event={"ID":"c614aac7-e486-4623-a8d2-a011541fdeee","Type":"ContainerStarted","Data":"2a8370f89fea1ed458729093094b767ec21a9d289233912baa69907a6927f00a"} Jul 28 17:34:58.693110 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.693063 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-67cb9" event={"ID":"7785bf5b-11ef-44ba-a530-f83e062952a9","Type":"ContainerStarted","Data":"c3c748d790318b19ba0629d6b27b7bedd47207ef11780376d896d13eeec633bc"} Jul 28 17:34:58.694807 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.694783 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" event={"ID":"0080a572-035a-4696-b249-18c75ec5e83c","Type":"ContainerStarted","Data":"b400fadcc7068848befc1b9c62a11de72f26f41c5000fb36e510d45c15b9ca3d"} Jul 28 17:34:58.696229 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.696207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7q7r" event={"ID":"3bcbad70-e788-408a-8f1b-15c7bffafcfe","Type":"ContainerStarted","Data":"8a388bef43d32448cf4d43dffae64c1f899453fe5e9b5f5180dd5a230951a7a8"} Jul 28 17:34:58.697850 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.697827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9jntc" event={"ID":"7c496d9d-d40c-4b43-920d-ceab761c6ad5","Type":"ContainerStarted","Data":"d812b145a8874f1ffc96bc240b6b523b3e828ac61630d541df9cfdb7c8901cf2"} Jul 28 17:34:58.699290 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.699269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" event={"ID":"51112e6a-064a-4a92-ad7f-bd511b166d9f","Type":"ContainerStarted","Data":"e0382fe4f10945bc6254ef672490d31c2d19dfc3442c6c399e1ca03814aff7da"} Jul 28 17:34:58.710331 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.710294 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-157.ec2.internal" podStartSLOduration=21.710281657 podStartE2EDuration="21.710281657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:40.661742626 +0000 UTC m=+5.634250182" watchObservedRunningTime="2026-07-28 17:34:58.710281657 +0000 UTC m=+23.682789207" Jul 28 17:34:58.728295 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.728255 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v6ldd" podStartSLOduration=3.490960567 podStartE2EDuration="22.728244151s" totalImagesPullingTime="19.237283584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:58.727960256 +0000 UTC m=+23.700467806" watchObservedRunningTime="2026-07-28 17:34:58.728244151 +0000 UTC m=+23.700751702" Jul 28 17:34:58.750834 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.750793 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l7q7r" podStartSLOduration=3.548427593 podStartE2EDuration="22.750781341s" totalImagesPullingTime="19.202353748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:58.750141337 +0000 UTC m=+23.722648887" watchObservedRunningTime="2026-07-28 17:34:58.750781341 +0000 UTC m=+23.723288944" Jul 28 17:34:58.764621 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.764579 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-prxq5" podStartSLOduration=3.5414874579999998 podStartE2EDuration="22.764567306s" totalImagesPullingTime="19.223079848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:58.764435962 +0000 UTC m=+23.736943514" watchObservedRunningTime="2026-07-28 17:34:58.764567306 +0000 UTC m=+23.737074856" Jul 28 17:34:58.778758 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.778703 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-67cb9" podStartSLOduration=3.568120629 podStartE2EDuration="22.778685987s" totalImagesPullingTime="19.210565358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:58.778454527 +0000 UTC m=+23.750962079" watchObservedRunningTime="2026-07-28 17:34:58.778685987 +0000 UTC m=+23.751193545" Jul 28 17:34:58.807637 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:58.807601 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9jntc" podStartSLOduration=3.530781444 podStartE2EDuration="22.80758872s" totalImagesPullingTime="19.276807276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:58.807418724 +0000 UTC m=+23.779926278" watchObservedRunningTime="2026-07-28 17:34:58.80758872 +0000 UTC m=+23.780096271" Jul 28 17:34:59.214062 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.214034 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 17:34:59.452272 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.452133 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:34:59.214054492Z","UUID":"23d4f720-3d71-4c02-866c-2248942d64c8","Handler":null,"Name":"","Endpoint":""} Jul 28 17:34:59.453956 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.453934 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 17:34:59.454090 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.453970 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 17:34:59.514165 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.514083 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:59.590300 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.590267 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:34:59.590466 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.590271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:34:59.590466 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:59.590406 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:34:59.590995 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:34:59.590488 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:34:59.705033 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.704995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" event={"ID":"0080a572-035a-4696-b249-18c75ec5e83c","Type":"ContainerStarted","Data":"e4c014790ea33cb8cf83fc6f2245446a4aa8ceba9baf49c02805b4e4e1b1539b"} Jul 28 17:34:59.706777 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.706687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mxfwr" event={"ID":"141276f1-330f-4de4-a16e-8521cc402881","Type":"ContainerStarted","Data":"7e7dcfc8ae48b5b8932e492c35ae9dd9a8efe91936b2b0720e6e21da7805f136"} Jul 28 17:34:59.724985 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.724897 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mxfwr" podStartSLOduration=4.521070728 podStartE2EDuration="23.724875896s" totalImagesPullingTime="19.203805168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:59.724581877 +0000 UTC m=+24.697089445" watchObservedRunningTime="2026-07-28 17:34:59.724875896 +0000 UTC m=+24.697383427" Jul 28 17:34:59.849399 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.849323 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:34:59.849995 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:34:59.849975 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:35:00.590287 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:00.590258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:00.590457 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:00.590433 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:00.712989 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:00.712951 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" event={"ID":"c614aac7-e486-4623-a8d2-a011541fdeee","Type":"ContainerStarted","Data":"d1a54f9903a128073fef84b1e6c3a2a2cee059fa51c7d9f761c55d32d22b8c03"} Jul 28 17:35:00.715353 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:00.715329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" event={"ID":"0080a572-035a-4696-b249-18c75ec5e83c","Type":"ContainerStarted","Data":"30f3b7666fcd31b35c3f1b73b8d84253d6cd3aadc0580bd4e85343fbeb320d4d"} Jul 28 17:35:00.716033 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:00.716011 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l7q7r" Jul 28 17:35:00.734388 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:00.734344 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92n7v" podStartSLOduration=3.261583362 podStartE2EDuration="24.734327625s" totalImagesPullingTime="21.472744263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:00.733351174 +0000 UTC m=+25.705858749" watchObservedRunningTime="2026-07-28 17:35:00.734327625 +0000 UTC m=+25.706835178" Jul 28 17:35:01.590304 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:01.590272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:01.590304 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:01.590294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:01.590642 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:01.590395 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:01.590642 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:01.590470 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:02.590645 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:02.590616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:02.591095 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:02.590753 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:03.590289 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:03.590096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:03.590446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:03.590108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:03.590446 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:03.590360 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:03.590446 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:03.590429 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:03.726407 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:03.726368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" event={"ID":"c614aac7-e486-4623-a8d2-a011541fdeee","Type":"ContainerStarted","Data":"8ed2d5803728a390f881f6816f8ded80cb9f8b275c8e12dbaba45d3fcd264b7d"} Jul 28 17:35:03.727308 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:03.726519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:35:03.727308 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:03.726579 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:35:03.728260 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:03.728224 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ffba58f-7387-4f1f-b44c-a143ba089097" containerID="3cbb1543e8a29edd2a94f42f3a700148829e1c01cca80546f4fc786e98733be7" exitCode=0 Jul 28 17:35:03.728354 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:03.728267 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" event={"ID":"8ffba58f-7387-4f1f-b44c-a143ba089097","Type":"ContainerDied","Data":"3cbb1543e8a29edd2a94f42f3a700148829e1c01cca80546f4fc786e98733be7"} Jul 28 17:35:03.742419 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:03.742400 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:35:03.752726 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:03.752690 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" podStartSLOduration=8.434742145 podStartE2EDuration="27.752676674s" totalImagesPullingTime="19.317934529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:03.751692224 +0000 UTC m=+28.724199775" watchObservedRunningTime="2026-07-28 17:35:03.752676674 +0000 UTC m=+28.725184219" Jul 28 17:35:04.590768 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:04.590573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:04.591034 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:04.590859 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:04.730470 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:04.730437 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:35:04.749321 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:04.749289 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:35:04.946667 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:04.946624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zp4hx"] Jul 28 17:35:04.946841 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:04.946762 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:04.946888 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:04.946848 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:04.950108 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:04.950079 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mq7v6"] Jul 28 17:35:04.950305 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:04.950236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:04.950385 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:04.950357 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:04.953945 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:04.953100 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7hqdl"] Jul 28 17:35:04.953945 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:04.953293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:04.953945 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:04.953424 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:05.733989 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:05.733908 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ffba58f-7387-4f1f-b44c-a143ba089097" containerID="4634c77c5652e6a1a850e3796aa94e7a920ba5a244fb42e8b9beaa6e265f439b" exitCode=0 Jul 28 17:35:05.734462 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:05.733998 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" event={"ID":"8ffba58f-7387-4f1f-b44c-a143ba089097","Type":"ContainerDied","Data":"4634c77c5652e6a1a850e3796aa94e7a920ba5a244fb42e8b9beaa6e265f439b"} Jul 28 17:35:06.590506 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:06.590470 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:06.590671 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:06.590573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:06.590671 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:06.590577 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:06.590671 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:06.590616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:06.590769 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:06.590697 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:06.590843 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:06.590772 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:07.741432 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:07.741247 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ffba58f-7387-4f1f-b44c-a143ba089097" containerID="e5fe5f6bb499b4976ed8e2945b0b7cd1c4ea494705e864de04e214fe5519ad93" exitCode=0 Jul 28 17:35:07.741805 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:07.741287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" event={"ID":"8ffba58f-7387-4f1f-b44c-a143ba089097","Type":"ContainerDied","Data":"e5fe5f6bb499b4976ed8e2945b0b7cd1c4ea494705e864de04e214fe5519ad93"} Jul 28 17:35:07.858922 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:07.858890 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-prxq5_de3de89b-e0c9-4347-b778-e7893bef0f5e/dns-node-resolver/0.log" Jul 28 17:35:08.429071 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:08.429002 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-67cb9_7785bf5b-11ef-44ba-a530-f83e062952a9/node-ca/0.log" Jul 28 17:35:08.590105 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:08.590066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:08.590353 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:08.590081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:08.590353 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:08.590236 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:08.590353 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:08.590082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:08.590353 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:08.590294 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:08.590353 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:08.590350 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:09.130673 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:09.130640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:09.131094 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:09.130789 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:35:09.131094 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:09.130854 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret podName:ff31c586-d5f6-44de-bf64-b6539584c26d nodeName:}" failed. No retries permitted until 2026-07-28 17:35:41.130837832 +0000 UTC m=+66.103345366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret") pod "global-pull-secret-syncer-zp4hx" (UID: "ff31c586-d5f6-44de-bf64-b6539584c26d") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:35:09.231982 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:09.231951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:09.232261 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:09.232001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:09.232261 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:09.232122 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:35:09.232261 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:09.232137 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:35:09.232261 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:09.232156 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:35:09.232261 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:09.232167 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lhrln for pod openshift-network-diagnostics/network-check-target-mq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:35:09.232261 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:09.232209 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs podName:c6998d3c-ff71-41f2-972c-8f5bfcee0614 nodeName:}" failed. No retries permitted until 2026-07-28 17:35:41.232188864 +0000 UTC m=+66.204696415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs") pod "network-metrics-daemon-7hqdl" (UID: "c6998d3c-ff71-41f2-972c-8f5bfcee0614") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:35:09.232261 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:09.232231 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln podName:e7d31075-9a8d-4306-9f59-c82ae381c980 nodeName:}" failed. No retries permitted until 2026-07-28 17:35:41.232216804 +0000 UTC m=+66.204724346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhrln" (UniqueName: "kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln") pod "network-check-target-mq7v6" (UID: "e7d31075-9a8d-4306-9f59-c82ae381c980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:35:10.590234 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:10.590199 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:10.590686 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:10.590244 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:10.590686 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:10.590244 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:10.590686 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:10.590322 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:10.590686 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:10.590423 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:10.590686 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:10.590506 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:11.753870 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:11.753838 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ffba58f-7387-4f1f-b44c-a143ba089097" containerID="39c36684b304c725b416cef869c83387235eed9ef21c541dde4275c6e4e526b1" exitCode=0 Jul 28 17:35:11.754241 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:11.753899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" event={"ID":"8ffba58f-7387-4f1f-b44c-a143ba089097","Type":"ContainerDied","Data":"39c36684b304c725b416cef869c83387235eed9ef21c541dde4275c6e4e526b1"} Jul 28 17:35:12.590392 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:12.590359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:12.590573 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:12.590359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:12.590573 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:12.590472 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:12.590573 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:12.590359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:12.590733 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:12.590535 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:12.590733 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:12.590655 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:12.758614 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:12.758582 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ffba58f-7387-4f1f-b44c-a143ba089097" containerID="aa9ea35859f965387b7ef8b358e7c6b8b84846a89a30ed610381d9f7c27e3f88" exitCode=0 Jul 28 17:35:12.759064 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:12.758630 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" event={"ID":"8ffba58f-7387-4f1f-b44c-a143ba089097","Type":"ContainerDied","Data":"aa9ea35859f965387b7ef8b358e7c6b8b84846a89a30ed610381d9f7c27e3f88"} Jul 28 17:35:13.763302 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:13.763269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" event={"ID":"8ffba58f-7387-4f1f-b44c-a143ba089097","Type":"ContainerStarted","Data":"95c77d7bbea870ca84930b3d164eecb63419fbaaff2d01e99a0cf93b9c6e7a18"} Jul 28 17:35:13.785680 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:13.785625 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tzk4n" podStartSLOduration=7.875809852 podStartE2EDuration="37.785609525s" totalImagesPullingTime="29.909799673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:13.784602264 +0000 UTC m=+38.757109824" watchObservedRunningTime="2026-07-28 17:35:13.785609525 +0000 UTC m=+38.758117077" Jul 28 17:35:14.590468 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:14.590442 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:14.590598 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:14.590471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:14.590598 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:14.590533 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:14.590598 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:14.590574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:14.590705 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:14.590662 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:14.590705 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:14.590696 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:16.590119 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:16.590077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:16.590119 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:16.590125 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:16.590566 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:16.590191 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:16.590566 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:16.590213 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:16.590566 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:16.590274 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:16.590566 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:16.590338 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:18.590382 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:18.590349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:18.590382 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:18.590385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:18.590794 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:18.590448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:18.590794 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:18.590461 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:18.590794 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:18.590527 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:18.590794 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:18.590570 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:20.589875 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:20.589849 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:20.590347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:20.589878 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:20.590347 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:20.589950 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:20.590347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:20.589993 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:20.590347 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:20.590060 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:20.590347 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:20.590113 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:22.590193 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:22.590152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:22.590562 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:22.590254 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:22.590562 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:22.590259 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:22.590562 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:22.590278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:22.590562 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:22.590340 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:22.590562 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:22.590415 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:24.589966 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:24.589933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:24.589966 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:24.589957 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:24.589966 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:24.589975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:24.590450 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:24.590060 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:24.590450 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:24.590187 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:24.590450 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:24.590264 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:26.590324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:26.590285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:26.590324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:26.590309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:26.590324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:26.590309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:26.590858 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:26.590392 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hqdl" podUID="c6998d3c-ff71-41f2-972c-8f5bfcee0614" Jul 28 17:35:26.590858 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:26.590489 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq7v6" podUID="e7d31075-9a8d-4306-9f59-c82ae381c980" Jul 28 17:35:26.590858 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:26.590593 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp4hx" podUID="ff31c586-d5f6-44de-bf64-b6539584c26d" Jul 28 17:35:27.355977 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.355949 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-157.ec2.internal" event="NodeReady" Jul 28 17:35:27.356135 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.356087 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 17:35:27.409822 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.409770 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8qb72"] Jul 28 17:35:27.432538 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.432510 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-shzch"] Jul 28 17:35:27.432689 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.432636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.435249 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.435225 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 17:35:27.435249 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.435243 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 17:35:27.435466 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.435225 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 17:35:27.435466 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.435302 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xj8mn\"" Jul 28 17:35:27.435466 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.435343 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 17:35:27.452881 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.452802 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shzch"] Jul 28 17:35:27.452881 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.452837 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8qb72"] Jul 28 17:35:27.453079 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.452938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shzch" Jul 28 17:35:27.456243 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.456216 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 17:35:27.456366 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.456278 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7mzg2\"" Jul 28 17:35:27.456366 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.456292 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 17:35:27.456553 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.456539 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 17:35:27.462703 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.462682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsl5k\" (UniqueName: \"kubernetes.io/projected/2158ecfe-9984-4ecb-9a13-7c02d3cad353-kube-api-access-fsl5k\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.462796 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.462733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2158ecfe-9984-4ecb-9a13-7c02d3cad353-crio-socket\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.462796 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.462755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2158ecfe-9984-4ecb-9a13-7c02d3cad353-data-volume\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.462796 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.462780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcg5s\" (UniqueName: \"kubernetes.io/projected/48f71842-cc3b-4b31-be38-a7687228407a-kube-api-access-fcg5s\") pod \"ingress-canary-shzch\" (UID: \"48f71842-cc3b-4b31-be38-a7687228407a\") " pod="openshift-ingress-canary/ingress-canary-shzch" Jul 28 17:35:27.462914 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.462800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2158ecfe-9984-4ecb-9a13-7c02d3cad353-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.462914 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.462816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f71842-cc3b-4b31-be38-a7687228407a-cert\") pod \"ingress-canary-shzch\" (UID: \"48f71842-cc3b-4b31-be38-a7687228407a\") " pod="openshift-ingress-canary/ingress-canary-shzch" Jul 28 17:35:27.462914 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.462838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2158ecfe-9984-4ecb-9a13-7c02d3cad353-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.510030 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.509994 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z6ph2"] Jul 28 17:35:27.533318 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.533274 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z6ph2"] Jul 28 17:35:27.533492 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.533412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.535785 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.535757 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 17:35:27.535938 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.535757 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 17:35:27.535938 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.535794 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxmvx\"" Jul 28 17:35:27.563880 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.563844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2158ecfe-9984-4ecb-9a13-7c02d3cad353-crio-socket\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.563880 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.563883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2158ecfe-9984-4ecb-9a13-7c02d3cad353-data-volume\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.564129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.563904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7npm\" (UniqueName: \"kubernetes.io/projected/df9b2c95-b7de-43dd-83c0-1f49969d5d25-kube-api-access-g7npm\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.564129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.563945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg5s\" (UniqueName: \"kubernetes.io/projected/48f71842-cc3b-4b31-be38-a7687228407a-kube-api-access-fcg5s\") pod \"ingress-canary-shzch\" (UID: \"48f71842-cc3b-4b31-be38-a7687228407a\") " pod="openshift-ingress-canary/ingress-canary-shzch" Jul 28 17:35:27.564129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.563963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df9b2c95-b7de-43dd-83c0-1f49969d5d25-config-volume\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.564129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.563981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2158ecfe-9984-4ecb-9a13-7c02d3cad353-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.564129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.564001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f71842-cc3b-4b31-be38-a7687228407a-cert\") pod \"ingress-canary-shzch\" (UID: \"48f71842-cc3b-4b31-be38-a7687228407a\") " pod="openshift-ingress-canary/ingress-canary-shzch" Jul 28 17:35:27.564129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.564019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2158ecfe-9984-4ecb-9a13-7c02d3cad353-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.564129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.564038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df9b2c95-b7de-43dd-83c0-1f49969d5d25-metrics-tls\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.564129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.564062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9b2c95-b7de-43dd-83c0-1f49969d5d25-tmp-dir\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.564129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.564057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2158ecfe-9984-4ecb-9a13-7c02d3cad353-crio-socket\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.564491 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.564137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsl5k\" (UniqueName: \"kubernetes.io/projected/2158ecfe-9984-4ecb-9a13-7c02d3cad353-kube-api-access-fsl5k\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.564491 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.564276 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2158ecfe-9984-4ecb-9a13-7c02d3cad353-data-volume\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.567818 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.567787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2158ecfe-9984-4ecb-9a13-7c02d3cad353-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.567948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.567933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f71842-cc3b-4b31-be38-a7687228407a-cert\") pod \"ingress-canary-shzch\" (UID: \"48f71842-cc3b-4b31-be38-a7687228407a\") " pod="openshift-ingress-canary/ingress-canary-shzch" Jul 28 17:35:27.572440 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.572414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsl5k\" (UniqueName: \"kubernetes.io/projected/2158ecfe-9984-4ecb-9a13-7c02d3cad353-kube-api-access-fsl5k\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.572831 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.572801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2158ecfe-9984-4ecb-9a13-7c02d3cad353-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qb72\" (UID: \"2158ecfe-9984-4ecb-9a13-7c02d3cad353\") " pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.575866 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.575847 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcg5s\" (UniqueName: \"kubernetes.io/projected/48f71842-cc3b-4b31-be38-a7687228407a-kube-api-access-fcg5s\") pod \"ingress-canary-shzch\" (UID: \"48f71842-cc3b-4b31-be38-a7687228407a\") " pod="openshift-ingress-canary/ingress-canary-shzch" Jul 28 17:35:27.664640 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.664604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7npm\" (UniqueName: \"kubernetes.io/projected/df9b2c95-b7de-43dd-83c0-1f49969d5d25-kube-api-access-g7npm\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.664640 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.664651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df9b2c95-b7de-43dd-83c0-1f49969d5d25-config-volume\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.665053 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.664672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df9b2c95-b7de-43dd-83c0-1f49969d5d25-metrics-tls\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.665053 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.664687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9b2c95-b7de-43dd-83c0-1f49969d5d25-tmp-dir\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.665115 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.665069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9b2c95-b7de-43dd-83c0-1f49969d5d25-tmp-dir\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.665282 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.665256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df9b2c95-b7de-43dd-83c0-1f49969d5d25-config-volume\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.667017 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.666997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df9b2c95-b7de-43dd-83c0-1f49969d5d25-metrics-tls\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.673115 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.673084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7npm\" (UniqueName: \"kubernetes.io/projected/df9b2c95-b7de-43dd-83c0-1f49969d5d25-kube-api-access-g7npm\") pod \"dns-default-z6ph2\" (UID: \"df9b2c95-b7de-43dd-83c0-1f49969d5d25\") " pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.742164 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.742065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8qb72" Jul 28 17:35:27.761259 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.761229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shzch" Jul 28 17:35:27.844307 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.844270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:27.935098 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.935076 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8qb72"] Jul 28 17:35:27.938215 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:27.938183 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2158ecfe_9984_4ecb_9a13_7c02d3cad353.slice/crio-0b4b527c4d63fb2bac8b27a07ed88a58885def72dad707bb60d2e4b96184f6dc WatchSource:0}: Error finding container 0b4b527c4d63fb2bac8b27a07ed88a58885def72dad707bb60d2e4b96184f6dc: Status 404 returned error can't find the container with id 0b4b527c4d63fb2bac8b27a07ed88a58885def72dad707bb60d2e4b96184f6dc Jul 28 17:35:27.949483 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.949459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shzch"] Jul 28 17:35:27.952909 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:27.952875 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f71842_cc3b_4b31_be38_a7687228407a.slice/crio-43208117ff56a2917d97f1eaf2e2682a10696f28870c724714c58e5892d28e06 WatchSource:0}: Error finding container 43208117ff56a2917d97f1eaf2e2682a10696f28870c724714c58e5892d28e06: Status 404 returned error can't find the container with id 43208117ff56a2917d97f1eaf2e2682a10696f28870c724714c58e5892d28e06 Jul 28 17:35:27.992355 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:27.992277 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z6ph2"] Jul 28 17:35:27.995341 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:27.995316 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9b2c95_b7de_43dd_83c0_1f49969d5d25.slice/crio-d23300ce36f9ae018b795257f8e78e9cebbaadf0449cca6153dc79975cf94467 WatchSource:0}: Error finding container d23300ce36f9ae018b795257f8e78e9cebbaadf0449cca6153dc79975cf94467: Status 404 returned error can't find the container with id d23300ce36f9ae018b795257f8e78e9cebbaadf0449cca6153dc79975cf94467 Jul 28 17:35:28.590401 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.590369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:28.590401 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.590399 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:28.590627 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.590400 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:28.592910 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.592879 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:35:28.592910 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.592899 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:35:28.593473 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.593430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:35:28.593606 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.593545 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z5ccl\"" Jul 28 17:35:28.593671 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.593620 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c22cz\"" Jul 28 17:35:28.593728 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.593715 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:35:28.798907 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.798864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qb72" event={"ID":"2158ecfe-9984-4ecb-9a13-7c02d3cad353","Type":"ContainerStarted","Data":"0b4b527c4d63fb2bac8b27a07ed88a58885def72dad707bb60d2e4b96184f6dc"} Jul 28 17:35:28.800872 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.800844 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z6ph2" event={"ID":"df9b2c95-b7de-43dd-83c0-1f49969d5d25","Type":"ContainerStarted","Data":"d23300ce36f9ae018b795257f8e78e9cebbaadf0449cca6153dc79975cf94467"} Jul 28 17:35:28.802476 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:28.802451 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shzch" event={"ID":"48f71842-cc3b-4b31-be38-a7687228407a","Type":"ContainerStarted","Data":"43208117ff56a2917d97f1eaf2e2682a10696f28870c724714c58e5892d28e06"} Jul 28 17:35:33.817518 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:33.817474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qb72" event={"ID":"2158ecfe-9984-4ecb-9a13-7c02d3cad353","Type":"ContainerStarted","Data":"2343408d64581c1757569de7687fbbe747e9f366efb9f3359165dd7a540f27ec"} Jul 28 17:35:33.819295 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:33.819270 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z6ph2" event={"ID":"df9b2c95-b7de-43dd-83c0-1f49969d5d25","Type":"ContainerStarted","Data":"b7515cc884183199a85576d1ceec64bf3a94346c43eaed354623f5e008a1ac0f"} Jul 28 17:35:33.819427 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:33.819320 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z6ph2" event={"ID":"df9b2c95-b7de-43dd-83c0-1f49969d5d25","Type":"ContainerStarted","Data":"1d708d3d7cb8750fbac161e27f198c29531130ba611ec331a72465d65342a2ff"} Jul 28 17:35:33.819427 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:33.819348 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:33.820752 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:33.820723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shzch" event={"ID":"48f71842-cc3b-4b31-be38-a7687228407a","Type":"ContainerStarted","Data":"22ebd8d6fb7bdc43e9247360fa31f20fc15a71991ffbdfe06daf11ac1c1b0009"} Jul 28 17:35:33.851158 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:33.851056 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z6ph2" podStartSLOduration=1.269990908 podStartE2EDuration="6.851036945s" totalImagesPullingTime="5.581046037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:33.836407203 +0000 UTC m=+58.808914783" watchObservedRunningTime="2026-07-28 17:35:33.851036945 +0000 UTC m=+58.823544473" Jul 28 17:35:33.851314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:33.851209 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-shzch" podStartSLOduration=1.2282167720000001 podStartE2EDuration="6.851203161s" totalImagesPullingTime="5.622986389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:33.85045353 +0000 UTC m=+58.822961092" watchObservedRunningTime="2026-07-28 17:35:33.851203161 +0000 UTC m=+58.823710711" Jul 28 17:35:36.758234 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:36.758204 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5jqv8" Jul 28 17:35:37.835382 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:37.835340 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qb72" event={"ID":"2158ecfe-9984-4ecb-9a13-7c02d3cad353","Type":"ContainerStarted","Data":"26672fc3c6f65e76fa9f5f13ce173d25749f9c0cfa48f0ee11cd38b4c3c03dde"} Jul 28 17:35:37.849628 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:37.849557 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv"] Jul 28 17:35:37.851464 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:37.851448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" Jul 28 17:35:37.853539 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:37.853518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 28 17:35:37.853637 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:37.853603 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cnkfb\"" Jul 28 17:35:37.861828 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:37.861808 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv"] Jul 28 17:35:37.949061 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:37.949021 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8aedc707-345f-4645-9935-c1afe6cd5d01-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-sbzkv\" (UID: \"8aedc707-345f-4645-9935-c1afe6cd5d01\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" Jul 28 17:35:38.049454 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:38.049420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8aedc707-345f-4645-9935-c1afe6cd5d01-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-sbzkv\" (UID: \"8aedc707-345f-4645-9935-c1afe6cd5d01\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" Jul 28 17:35:38.052100 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:38.052071 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8aedc707-345f-4645-9935-c1afe6cd5d01-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-sbzkv\" (UID: \"8aedc707-345f-4645-9935-c1afe6cd5d01\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" Jul 28 17:35:38.160341 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:38.160304 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" Jul 28 17:35:38.305376 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:38.305345 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv"] Jul 28 17:35:38.308413 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:38.308386 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aedc707_345f_4645_9935_c1afe6cd5d01.slice/crio-f86961b2888e5b897c2a17e37adb04794785d68db1d5d7b39030498261f7a68f WatchSource:0}: Error finding container f86961b2888e5b897c2a17e37adb04794785d68db1d5d7b39030498261f7a68f: Status 404 returned error can't find the container with id f86961b2888e5b897c2a17e37adb04794785d68db1d5d7b39030498261f7a68f Jul 28 17:35:38.840516 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:38.840467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" event={"ID":"8aedc707-345f-4645-9935-c1afe6cd5d01","Type":"ContainerStarted","Data":"f86961b2888e5b897c2a17e37adb04794785d68db1d5d7b39030498261f7a68f"} Jul 28 17:35:39.845735 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:39.845653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" event={"ID":"8aedc707-345f-4645-9935-c1afe6cd5d01","Type":"ContainerStarted","Data":"a6d6bd3265a49c8a103a53d5debe3320430d14c0e98e7bca4960db0e485e3ede"} Jul 28 17:35:39.846215 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:39.845854 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" Jul 28 17:35:39.852364 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:39.852326 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" Jul 28 17:35:39.863118 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:39.863061 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sbzkv" podStartSLOduration=1.772581561 podStartE2EDuration="2.863043544s" totalImagesPullingTime="1.090461983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:39.862462184 +0000 UTC m=+64.834969738" watchObservedRunningTime="2026-07-28 17:35:39.863043544 +0000 UTC m=+64.835551093" Jul 28 17:35:40.930017 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:40.929983 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-24f2v"] Jul 28 17:35:40.947940 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:40.947911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-24f2v"] Jul 28 17:35:40.948093 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:40.948027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:40.950353 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:40.950316 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 17:35:40.950353 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:40.950327 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 17:35:40.951399 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:40.951100 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 17:35:40.951399 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:40.951114 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 17:35:40.951399 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:40.951205 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 17:35:40.951632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:40.951439 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fnhmj\"" Jul 28 17:35:41.077393 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.077359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.077551 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.077460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.077551 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.077485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4zf\" (UniqueName: \"kubernetes.io/projected/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-kube-api-access-5r4zf\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.077551 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.077545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.178254 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.178228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.178368 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.178270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:41.178368 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.178297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.178475 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.178413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4zf\" (UniqueName: \"kubernetes.io/projected/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-kube-api-access-5r4zf\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.178525 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.178490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.178871 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.178852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.180503 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.180458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.180579 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.180562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.180832 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.180819 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:35:41.187449 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.187427 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4zf\" (UniqueName: \"kubernetes.io/projected/f0af88e5-3201-45a1-8ac3-fc51cdc8914f-kube-api-access-5r4zf\") pod \"prometheus-operator-84d69f794d-24f2v\" (UID: \"f0af88e5-3201-45a1-8ac3-fc51cdc8914f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.191013 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.190991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff31c586-d5f6-44de-bf64-b6539584c26d-original-pull-secret\") pod \"global-pull-secret-syncer-zp4hx\" (UID: \"ff31c586-d5f6-44de-bf64-b6539584c26d\") " pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:41.213823 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.213794 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp4hx" Jul 28 17:35:41.260166 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.260124 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" Jul 28 17:35:41.279662 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.279608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:41.279826 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.279684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:41.281998 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.281968 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:35:41.282162 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.282018 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:35:41.292977 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.292844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6998d3c-ff71-41f2-972c-8f5bfcee0614-metrics-certs\") pod \"network-metrics-daemon-7hqdl\" (UID: \"c6998d3c-ff71-41f2-972c-8f5bfcee0614\") " pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:41.293457 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.293207 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:35:41.303680 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.303618 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrln\" (UniqueName: \"kubernetes.io/projected/e7d31075-9a8d-4306-9f59-c82ae381c980-kube-api-access-lhrln\") pod \"network-check-target-mq7v6\" (UID: \"e7d31075-9a8d-4306-9f59-c82ae381c980\") " pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:41.342654 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.342626 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zp4hx"] Jul 28 17:35:41.345622 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:41.345599 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff31c586_d5f6_44de_bf64_b6539584c26d.slice/crio-ebcfde6c900ee624f7f211a4dbfbd35203abde65f80582d8446fd42f1197aded WatchSource:0}: Error finding container ebcfde6c900ee624f7f211a4dbfbd35203abde65f80582d8446fd42f1197aded: Status 404 returned error can't find the container with id ebcfde6c900ee624f7f211a4dbfbd35203abde65f80582d8446fd42f1197aded Jul 28 17:35:41.387746 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.387719 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-24f2v"] Jul 28 17:35:41.390996 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:41.390970 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0af88e5_3201_45a1_8ac3_fc51cdc8914f.slice/crio-fde48d79d243a694c75f6baa2544619f3ab641bd3728f68e732c77e1a428c826 WatchSource:0}: Error finding container fde48d79d243a694c75f6baa2544619f3ab641bd3728f68e732c77e1a428c826: Status 404 returned error can't find the container with id fde48d79d243a694c75f6baa2544619f3ab641bd3728f68e732c77e1a428c826 Jul 28 17:35:41.503874 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.503791 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c22cz\"" Jul 28 17:35:41.511372 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.511347 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z5ccl\"" Jul 28 17:35:41.511440 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.511419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:41.519968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.519942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hqdl" Jul 28 17:35:41.642877 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.642711 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mq7v6"] Jul 28 17:35:41.645157 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:41.645128 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d31075_9a8d_4306_9f59_c82ae381c980.slice/crio-88cd5de23b6924a08d37e60e06a906ddc1a2ce6e3d4e5653030712e7fad15c3b WatchSource:0}: Error finding container 88cd5de23b6924a08d37e60e06a906ddc1a2ce6e3d4e5653030712e7fad15c3b: Status 404 returned error can't find the container with id 88cd5de23b6924a08d37e60e06a906ddc1a2ce6e3d4e5653030712e7fad15c3b Jul 28 17:35:41.655342 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.655317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7hqdl"] Jul 28 17:35:41.658072 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:41.658052 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6998d3c_ff71_41f2_972c_8f5bfcee0614.slice/crio-6c730fe11238eb52e4d546769b1cd4d035e595aa5278dd4003075dcb650d5ac7 WatchSource:0}: Error finding container 6c730fe11238eb52e4d546769b1cd4d035e595aa5278dd4003075dcb650d5ac7: Status 404 returned error can't find the container with id 6c730fe11238eb52e4d546769b1cd4d035e595aa5278dd4003075dcb650d5ac7 Jul 28 17:35:41.855533 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.855454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qb72" event={"ID":"2158ecfe-9984-4ecb-9a13-7c02d3cad353","Type":"ContainerStarted","Data":"fbabd60dff41c745b3e473b7e28c1c3a17c8db16d1ce816a2e02f9ef333a29e9"} Jul 28 17:35:41.856917 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.856881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zp4hx" event={"ID":"ff31c586-d5f6-44de-bf64-b6539584c26d","Type":"ContainerStarted","Data":"ebcfde6c900ee624f7f211a4dbfbd35203abde65f80582d8446fd42f1197aded"} Jul 28 17:35:41.858099 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.858070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7hqdl" event={"ID":"c6998d3c-ff71-41f2-972c-8f5bfcee0614","Type":"ContainerStarted","Data":"6c730fe11238eb52e4d546769b1cd4d035e595aa5278dd4003075dcb650d5ac7"} Jul 28 17:35:41.859777 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.859753 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mq7v6" event={"ID":"e7d31075-9a8d-4306-9f59-c82ae381c980","Type":"ContainerStarted","Data":"88cd5de23b6924a08d37e60e06a906ddc1a2ce6e3d4e5653030712e7fad15c3b"} Jul 28 17:35:41.861002 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.860968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" event={"ID":"f0af88e5-3201-45a1-8ac3-fc51cdc8914f","Type":"ContainerStarted","Data":"fde48d79d243a694c75f6baa2544619f3ab641bd3728f68e732c77e1a428c826"} Jul 28 17:35:41.873770 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:41.873715 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8qb72" podStartSLOduration=1.779831194 podStartE2EDuration="14.873696942s" totalImagesPullingTime="13.093865748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:35:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:41.873252645 +0000 UTC m=+66.845760196" watchObservedRunningTime="2026-07-28 17:35:41.873696942 +0000 UTC m=+66.846204518" Jul 28 17:35:42.060664 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.060633 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-49w5r"] Jul 28 17:35:42.062559 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.062528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-49w5r" Jul 28 17:35:42.064989 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.064811 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7xdf8\"" Jul 28 17:35:42.065195 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.065157 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 17:35:42.065396 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.065281 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 17:35:42.074041 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.073981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-49w5r"] Jul 28 17:35:42.186722 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.186641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27tdk\" (UniqueName: \"kubernetes.io/projected/7bf7b156-f793-493b-9f1a-91afef4fb73f-kube-api-access-27tdk\") pod \"downloads-7c97dc4555-49w5r\" (UID: \"7bf7b156-f793-493b-9f1a-91afef4fb73f\") " pod="openshift-console/downloads-7c97dc4555-49w5r" Jul 28 17:35:42.287678 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.287611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27tdk\" (UniqueName: \"kubernetes.io/projected/7bf7b156-f793-493b-9f1a-91afef4fb73f-kube-api-access-27tdk\") pod \"downloads-7c97dc4555-49w5r\" (UID: \"7bf7b156-f793-493b-9f1a-91afef4fb73f\") " pod="openshift-console/downloads-7c97dc4555-49w5r" Jul 28 17:35:42.304829 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.304738 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27tdk\" (UniqueName: \"kubernetes.io/projected/7bf7b156-f793-493b-9f1a-91afef4fb73f-kube-api-access-27tdk\") pod \"downloads-7c97dc4555-49w5r\" (UID: \"7bf7b156-f793-493b-9f1a-91afef4fb73f\") " pod="openshift-console/downloads-7c97dc4555-49w5r" Jul 28 17:35:42.376936 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.376846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-49w5r" Jul 28 17:35:42.560435 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:42.560363 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-49w5r"] Jul 28 17:35:42.867938 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:42.867867 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf7b156_f793_493b_9f1a_91afef4fb73f.slice/crio-f3cf15c1e5c627f2fc1a850de6f1a894bfaa45c5fb3ab3e79f5fa5cb56229206 WatchSource:0}: Error finding container f3cf15c1e5c627f2fc1a850de6f1a894bfaa45c5fb3ab3e79f5fa5cb56229206: Status 404 returned error can't find the container with id f3cf15c1e5c627f2fc1a850de6f1a894bfaa45c5fb3ab3e79f5fa5cb56229206 Jul 28 17:35:43.827439 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:43.826483 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z6ph2" Jul 28 17:35:43.880498 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:43.879633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7hqdl" event={"ID":"c6998d3c-ff71-41f2-972c-8f5bfcee0614","Type":"ContainerStarted","Data":"31f1c5cdb49169f3f02256ce30c4e9191c929d36854272d334df84e6211f8478"} Jul 28 17:35:43.880498 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:43.879735 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7hqdl" event={"ID":"c6998d3c-ff71-41f2-972c-8f5bfcee0614","Type":"ContainerStarted","Data":"d340d12f96c34ea4952e31cdbfb7e0bf3377cc5c0bd648a832d315e62b388b30"} Jul 28 17:35:43.885075 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:43.885020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-49w5r" event={"ID":"7bf7b156-f793-493b-9f1a-91afef4fb73f","Type":"ContainerStarted","Data":"f3cf15c1e5c627f2fc1a850de6f1a894bfaa45c5fb3ab3e79f5fa5cb56229206"} Jul 28 17:35:43.888712 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:43.888186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" event={"ID":"f0af88e5-3201-45a1-8ac3-fc51cdc8914f","Type":"ContainerStarted","Data":"db1ece54ce193a64d10c18bcff3b6b9c0ad92c13d992603eba5eb6e4c1d41eab"} Jul 28 17:35:43.888712 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:43.888237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" event={"ID":"f0af88e5-3201-45a1-8ac3-fc51cdc8914f","Type":"ContainerStarted","Data":"3f68eeb10c2f81e210dce84d2e29aae2428fbaa471cbc9e78506db35faa0d6e1"} Jul 28 17:35:43.901019 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:43.900262 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7hqdl" podStartSLOduration=66.351129894 podStartE2EDuration="1m7.900243586s" totalImagesPullingTime="1.549113692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:43.899458733 +0000 UTC m=+68.871966283" watchObservedRunningTime="2026-07-28 17:35:43.900243586 +0000 UTC m=+68.872751138" Jul 28 17:35:43.916560 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:43.916509 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-24f2v" podStartSLOduration=2.103354709 podStartE2EDuration="3.91649055s" totalImagesPullingTime="1.813135841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:43.915814629 +0000 UTC m=+68.888322205" watchObservedRunningTime="2026-07-28 17:35:43.91649055 +0000 UTC m=+68.888998122" Jul 28 17:35:46.290560 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.290522 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92"] Jul 28 17:35:46.293202 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.293167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.295362 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.295336 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 17:35:46.295485 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.295409 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 17:35:46.295696 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.295677 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g5kmg\"" Jul 28 17:35:46.303511 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.303489 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92"] Jul 28 17:35:46.324938 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.324868 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6zp92"] Jul 28 17:35:46.327371 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.327343 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh"] Jul 28 17:35:46.327971 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.327948 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.329621 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.329582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.331189 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.331141 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 17:35:46.331927 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.331905 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 17:35:46.332586 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.330362 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 17:35:46.333040 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.333016 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p6zg4\"" Jul 28 17:35:46.336770 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.336748 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 17:35:46.336969 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.336948 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nzhtc\"" Jul 28 17:35:46.337053 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.337033 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 17:35:46.337243 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.336827 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 17:35:46.339191 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.339150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh"] Jul 28 17:35:46.421366 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-textfile\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.421366 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.421623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.421623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9xf\" (UniqueName: \"kubernetes.io/projected/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-api-access-nw9xf\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.421623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-tls\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.421623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28f7079e-208d-4aea-ad43-eae93dffef7d-sys\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.421623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.421623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.421915 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24s8w\" (UniqueName: \"kubernetes.io/projected/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-kube-api-access-24s8w\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.421915 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.421915 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.421915 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.421915 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.421915 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlkn8\" (UniqueName: \"kubernetes.io/projected/28f7079e-208d-4aea-ad43-eae93dffef7d-kube-api-access-rlkn8\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.421915 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.421915 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-wtmp\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.422345 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/28f7079e-208d-4aea-ad43-eae93dffef7d-root\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.422345 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.421949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28f7079e-208d-4aea-ad43-eae93dffef7d-metrics-client-ca\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.422345 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.422000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.523340 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.523301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.523521 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.523362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-textfile\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.523521 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.523383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.523521 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.523403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.523521 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.523426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9xf\" (UniqueName: \"kubernetes.io/projected/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-api-access-nw9xf\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.523808 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.523780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-textfile\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.523945 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.523927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-tls\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.524015 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.523974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28f7079e-208d-4aea-ad43-eae93dffef7d-sys\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.524015 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.524115 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524021 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.524115 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.524115 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:46.524040 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 17:35:46.524115 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24s8w\" (UniqueName: \"kubernetes.io/projected/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-kube-api-access-24s8w\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.524115 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.524351 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:46.524145 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-tls podName:28f7079e-208d-4aea-ad43-eae93dffef7d nodeName:}" failed. No retries permitted until 2026-07-28 17:35:47.024123542 +0000 UTC m=+71.996631083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-tls") pod "node-exporter-6zp92" (UID: "28f7079e-208d-4aea-ad43-eae93dffef7d") : secret "node-exporter-tls" not found Jul 28 17:35:46.524351 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524152 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28f7079e-208d-4aea-ad43-eae93dffef7d-sys\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.524351 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.524351 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.524351 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.524351 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlkn8\" (UniqueName: \"kubernetes.io/projected/28f7079e-208d-4aea-ad43-eae93dffef7d-kube-api-access-rlkn8\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.524351 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.524693 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-wtmp\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.524693 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/28f7079e-208d-4aea-ad43-eae93dffef7d-root\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.524693 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28f7079e-208d-4aea-ad43-eae93dffef7d-metrics-client-ca\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.524848 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.524848 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.524968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28f7079e-208d-4aea-ad43-eae93dffef7d-metrics-client-ca\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.524968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.524885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.525237 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.525110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-wtmp\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.525237 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.525197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/28f7079e-208d-4aea-ad43-eae93dffef7d-root\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.525465 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.525442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.526468 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.526446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.526565 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.526454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.527478 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.527457 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.527600 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.527578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.528154 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.528132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.532632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.532611 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9xf\" (UniqueName: \"kubernetes.io/projected/ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2-kube-api-access-nw9xf\") pod \"kube-state-metrics-56d9f944b4-6g7nh\" (UID: \"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:46.534120 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.534102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlkn8\" (UniqueName: \"kubernetes.io/projected/28f7079e-208d-4aea-ad43-eae93dffef7d-kube-api-access-rlkn8\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:46.534239 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.534198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24s8w\" (UniqueName: \"kubernetes.io/projected/f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da-kube-api-access-24s8w\") pod \"openshift-state-metrics-b8f49f5c6-jzn92\" (UID: \"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.603894 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.603791 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" Jul 28 17:35:46.650799 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:46.650762 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" Jul 28 17:35:47.028297 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.028159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-tls\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:47.032198 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.032073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28f7079e-208d-4aea-ad43-eae93dffef7d-node-exporter-tls\") pod \"node-exporter-6zp92\" (UID: \"28f7079e-208d-4aea-ad43-eae93dffef7d\") " pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:47.068568 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.068543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92"] Jul 28 17:35:47.071351 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:47.071069 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dc4332_8ef9_41f0_83d0_d2cf40ceb1da.slice/crio-deb3756e3f8493387986a980b869789d7a4a9bda7cadb456e4fc07bab0f52a90 WatchSource:0}: Error finding container deb3756e3f8493387986a980b869789d7a4a9bda7cadb456e4fc07bab0f52a90: Status 404 returned error can't find the container with id deb3756e3f8493387986a980b869789d7a4a9bda7cadb456e4fc07bab0f52a90 Jul 28 17:35:47.078265 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.078238 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh"] Jul 28 17:35:47.083071 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:47.083015 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1fe921_33ad_4e0f_bfdb_8af1a6cc9fa2.slice/crio-69aeca646d16901fb220bfdc66badaa690bfebc47d8cef846a4ba337ae752667 WatchSource:0}: Error finding container 69aeca646d16901fb220bfdc66badaa690bfebc47d8cef846a4ba337ae752667: Status 404 returned error can't find the container with id 69aeca646d16901fb220bfdc66badaa690bfebc47d8cef846a4ba337ae752667 Jul 28 17:35:47.243723 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.243694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6zp92" Jul 28 17:35:47.257096 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:47.257060 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f7079e_208d_4aea_ad43_eae93dffef7d.slice/crio-1dc4c41f30684d4dd00e33d0877f0fcaca18f6190c8506a8e44494c217f23c18 WatchSource:0}: Error finding container 1dc4c41f30684d4dd00e33d0877f0fcaca18f6190c8506a8e44494c217f23c18: Status 404 returned error can't find the container with id 1dc4c41f30684d4dd00e33d0877f0fcaca18f6190c8506a8e44494c217f23c18 Jul 28 17:35:47.386312 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.386212 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:35:47.389660 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.389639 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.392706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392061 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 17:35:47.392706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392114 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 17:35:47.392706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392149 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 17:35:47.392706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392387 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 17:35:47.392706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392508 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 17:35:47.392706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392388 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 17:35:47.392706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392392 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hw99m\"" Jul 28 17:35:47.392706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392657 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 17:35:47.393207 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392775 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 17:35:47.393207 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.392858 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 17:35:47.405867 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.405843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:35:47.432296 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432469 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432469 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432469 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432469 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-web-config\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432678 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432678 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432678 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432678 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432678 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbbc8\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-kube-api-access-zbbc8\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-out\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.432934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.432774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533280 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533280 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbbc8\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-kube-api-access-zbbc8\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533496 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-out\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533496 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533496 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533496 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533496 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533725 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:35:47.533538 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-trusted-ca-bundle podName:0b709b3c-28c2-4a16-9f54-2b2a69c2d36a nodeName:}" failed. No retries permitted until 2026-07-28 17:35:48.033509688 +0000 UTC m=+73.006017219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a") : configmap references non-existent config key: ca-bundle.crt Jul 28 17:35:47.533725 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533725 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533725 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-web-config\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533725 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.533725 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.534035 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.534035 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.533796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.535364 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.534585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.536701 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.536652 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.536906 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.536863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.537011 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.536946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.537465 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.537243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-web-config\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.537465 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.537399 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-out\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.537465 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.537426 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.538706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.538664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.538799 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.538714 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.539394 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.539340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.546008 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.545976 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbbc8\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-kube-api-access-zbbc8\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:47.913494 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.913416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zp4hx" event={"ID":"ff31c586-d5f6-44de-bf64-b6539584c26d","Type":"ContainerStarted","Data":"fd54ce6cdf0f9f440e7c43f22b80fde783756c16f464ae854892add35e03cd16"} Jul 28 17:35:47.915838 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.915802 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" event={"ID":"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da","Type":"ContainerStarted","Data":"0c54f4391d51754357ecf31448b351c97726603db449bbf336b780f63d6bfdec"} Jul 28 17:35:47.916007 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.915869 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" event={"ID":"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da","Type":"ContainerStarted","Data":"a64f70812e47d41d804caab3b29da42bab3f7fd0c55a4ade6c62e24672b57497"} Jul 28 17:35:47.916007 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.915901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" event={"ID":"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da","Type":"ContainerStarted","Data":"deb3756e3f8493387986a980b869789d7a4a9bda7cadb456e4fc07bab0f52a90"} Jul 28 17:35:47.917796 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.917667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mq7v6" event={"ID":"e7d31075-9a8d-4306-9f59-c82ae381c980","Type":"ContainerStarted","Data":"2426a2dab847911221b3b56a6d8dc1052afd32ed6a91dd0a40ad1bc7b586e584"} Jul 28 17:35:47.917796 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.917724 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:35:47.919238 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.919206 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zp92" event={"ID":"28f7079e-208d-4aea-ad43-eae93dffef7d","Type":"ContainerStarted","Data":"1dc4c41f30684d4dd00e33d0877f0fcaca18f6190c8506a8e44494c217f23c18"} Jul 28 17:35:47.920555 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.920533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" event={"ID":"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2","Type":"ContainerStarted","Data":"69aeca646d16901fb220bfdc66badaa690bfebc47d8cef846a4ba337ae752667"} Jul 28 17:35:47.946011 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.945492 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zp4hx" podStartSLOduration=66.365081801 podStartE2EDuration="1m11.945469457s" totalImagesPullingTime="5.580387656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:47.928546943 +0000 UTC m=+72.901054505" watchObservedRunningTime="2026-07-28 17:35:47.945469457 +0000 UTC m=+72.917977005" Jul 28 17:35:47.946235 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:47.946080 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mq7v6" podStartSLOduration=66.702838551 podStartE2EDuration="1m11.946067655s" totalImagesPullingTime="5.243229104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:47.945578271 +0000 UTC m=+72.918085819" watchObservedRunningTime="2026-07-28 17:35:47.946067655 +0000 UTC m=+72.918575208" Jul 28 17:35:48.040198 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.039316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:48.040947 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.040909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:48.303444 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.303332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:35:48.457064 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.456974 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws"] Jul 28 17:35:48.460658 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.460633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.463512 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.463384 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 17:35:48.463846 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.463709 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 17:35:48.463846 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.463746 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x72jq\"" Jul 28 17:35:48.465503 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.464161 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 17:35:48.465503 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.464353 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 17:35:48.465503 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.464755 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 17:35:48.465503 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.465338 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7h1ld5i11s6u3\"" Jul 28 17:35:48.479991 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.479892 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws"] Jul 28 17:35:48.545746 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.545708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.545934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.545764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.545934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.545851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.545934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.545893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4wgd\" (UniqueName: \"kubernetes.io/projected/c093872e-6524-4834-9b1f-1e2dbd258945-kube-api-access-f4wgd\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.545934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.545923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-grpc-tls\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.546158 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.545958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-tls\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.546158 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.546012 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c093872e-6524-4834-9b1f-1e2dbd258945-metrics-client-ca\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.546158 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.546046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.647221 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.647144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.647397 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.647235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.647397 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.647273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.647397 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.647296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4wgd\" (UniqueName: \"kubernetes.io/projected/c093872e-6524-4834-9b1f-1e2dbd258945-kube-api-access-f4wgd\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.647397 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.647320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-grpc-tls\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.647397 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.647348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-tls\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.647397 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.647386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c093872e-6524-4834-9b1f-1e2dbd258945-metrics-client-ca\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.647688 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.647417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.648191 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.648143 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c093872e-6524-4834-9b1f-1e2dbd258945-metrics-client-ca\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.650265 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.650221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.651271 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.650771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-tls\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.651271 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.650937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.651271 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.651232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.651463 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.651313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.652396 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.652376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c093872e-6524-4834-9b1f-1e2dbd258945-secret-grpc-tls\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.657197 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.657134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4wgd\" (UniqueName: \"kubernetes.io/projected/c093872e-6524-4834-9b1f-1e2dbd258945-kube-api-access-f4wgd\") pod \"thanos-querier-67f7f7b7b8-fq4ws\" (UID: \"c093872e-6524-4834-9b1f-1e2dbd258945\") " pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:48.773848 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:48.773806 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:49.167990 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.167957 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:35:49.173127 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:49.173083 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b709b3c_28c2_4a16_9f54_2b2a69c2d36a.slice/crio-f79e682d5df917bc9ab8a13e2d6622db331ba79f48cecd11091090b9d7600378 WatchSource:0}: Error finding container f79e682d5df917bc9ab8a13e2d6622db331ba79f48cecd11091090b9d7600378: Status 404 returned error can't find the container with id f79e682d5df917bc9ab8a13e2d6622db331ba79f48cecd11091090b9d7600378 Jul 28 17:35:49.185093 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.185004 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws"] Jul 28 17:35:49.189165 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:49.189130 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc093872e_6524_4834_9b1f_1e2dbd258945.slice/crio-1029fe06ef0956552ed30bb5816083d60dec9cb4286d50db178003bb7f24ecdd WatchSource:0}: Error finding container 1029fe06ef0956552ed30bb5816083d60dec9cb4286d50db178003bb7f24ecdd: Status 404 returned error can't find the container with id 1029fe06ef0956552ed30bb5816083d60dec9cb4286d50db178003bb7f24ecdd Jul 28 17:35:49.933820 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.933664 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" event={"ID":"c093872e-6524-4834-9b1f-1e2dbd258945","Type":"ContainerStarted","Data":"1029fe06ef0956552ed30bb5816083d60dec9cb4286d50db178003bb7f24ecdd"} Jul 28 17:35:49.940098 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.939116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" event={"ID":"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2","Type":"ContainerStarted","Data":"959867d1f3ed239f5c45db68345b13bb436c1bcf346aa5c036c6db0fc4eac104"} Jul 28 17:35:49.940098 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.939194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" event={"ID":"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2","Type":"ContainerStarted","Data":"2fc0beb82380bed01c2276a3c4d9492edb9dce32194352c1921bf7c60941ecf8"} Jul 28 17:35:49.940098 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.939226 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" event={"ID":"ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2","Type":"ContainerStarted","Data":"20bbd0ab1c0f4ba664748e51e9f7caa5100e5aef7cb26397b8f4ba73e7b65695"} Jul 28 17:35:49.943192 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.943151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" event={"ID":"f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da","Type":"ContainerStarted","Data":"c2f8e313e5849a7a96b8ab533ce6e2cacf642da248b8bce2b1826269b71223f1"} Jul 28 17:35:49.946005 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.945842 2576 generic.go:356] "Generic (PLEG): container finished" podID="28f7079e-208d-4aea-ad43-eae93dffef7d" containerID="88f7eb45d2a96e34ea1529955d47307c3d91648ff154005026f65a71bdcb32c6" exitCode=0 Jul 28 17:35:49.946005 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.945953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zp92" event={"ID":"28f7079e-208d-4aea-ad43-eae93dffef7d","Type":"ContainerDied","Data":"88f7eb45d2a96e34ea1529955d47307c3d91648ff154005026f65a71bdcb32c6"} Jul 28 17:35:49.949096 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.948963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerStarted","Data":"f79e682d5df917bc9ab8a13e2d6622db331ba79f48cecd11091090b9d7600378"} Jul 28 17:35:49.962528 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.961526 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6g7nh" podStartSLOduration=2.001708638 podStartE2EDuration="3.96150717s" totalImagesPullingTime="1.959798532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:49.96007449 +0000 UTC m=+74.932582040" watchObservedRunningTime="2026-07-28 17:35:49.96150717 +0000 UTC m=+74.934014724" Jul 28 17:35:49.982019 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:49.981515 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jzn92" podStartSLOduration=2.193120857 podStartE2EDuration="3.981496502s" totalImagesPullingTime="1.788375645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:49.980762356 +0000 UTC m=+74.953269907" watchObservedRunningTime="2026-07-28 17:35:49.981496502 +0000 UTC m=+74.954004054" Jul 28 17:35:50.705958 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.705924 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f947b7b5b-qdrhs"] Jul 28 17:35:50.711532 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.711504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.713711 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.713679 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 28 17:35:50.713844 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.713750 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8vqp3g8ke6o9p\"" Jul 28 17:35:50.714600 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.714487 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 17:35:50.714665 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.714641 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 28 17:35:50.714774 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.714725 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 28 17:35:50.714884 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.714864 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-87295\"" Jul 28 17:35:50.718786 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.718729 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f947b7b5b-qdrhs"] Jul 28 17:35:50.769158 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.769125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcznq\" (UniqueName: \"kubernetes.io/projected/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-kube-api-access-xcznq\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.769313 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.769199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-audit-log\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.769313 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.769230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-secret-metrics-server-tls\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.769313 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.769245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-client-ca-bundle\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.769313 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.769278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-secret-metrics-server-client-certs\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.769313 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.769302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-metrics-server-audit-profiles\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.769547 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.769350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.870206 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.869966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-audit-log\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.870206 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.870016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-secret-metrics-server-tls\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.870206 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.870042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-client-ca-bundle\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.870206 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.870093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-secret-metrics-server-client-certs\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.870206 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.870155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-metrics-server-audit-profiles\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.870555 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.870244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.870555 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.870324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcznq\" (UniqueName: \"kubernetes.io/projected/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-kube-api-access-xcznq\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.870555 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.870434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-audit-log\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.871004 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.870981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.871284 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.871258 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-metrics-server-audit-profiles\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.873101 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.873061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-secret-metrics-server-tls\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.873487 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.873465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-client-ca-bundle\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.873739 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.873718 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-secret-metrics-server-client-certs\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.879816 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.879796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcznq\" (UniqueName: \"kubernetes.io/projected/42744d32-298e-42d4-9cc7-5b4b38f1dbf2-kube-api-access-xcznq\") pod \"metrics-server-7f947b7b5b-qdrhs\" (UID: \"42744d32-298e-42d4-9cc7-5b4b38f1dbf2\") " pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:50.957835 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.957533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zp92" event={"ID":"28f7079e-208d-4aea-ad43-eae93dffef7d","Type":"ContainerStarted","Data":"c3b17e628b4822d79d285ba6c6c6157b3ebb2be75fbc23c8ceeeb8bdeaa5d78a"} Jul 28 17:35:50.957835 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.957600 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zp92" event={"ID":"28f7079e-208d-4aea-ad43-eae93dffef7d","Type":"ContainerStarted","Data":"b0369c1a5a89fe20edebb2eea345077f2f8e39d85b144f3d9f448db3de92c5cb"} Jul 28 17:35:50.960045 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.960014 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerID="4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4" exitCode=0 Jul 28 17:35:50.960268 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.960133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerDied","Data":"4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4"} Jul 28 17:35:50.977050 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:50.976656 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6zp92" podStartSLOduration=3.141108907 podStartE2EDuration="4.976636304s" totalImagesPullingTime="1.7389991s" totalInitContainerRuntime="96.528297ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:50.976391725 +0000 UTC m=+75.948899302" watchObservedRunningTime="2026-07-28 17:35:50.976636304 +0000 UTC m=+75.949143854" Jul 28 17:35:51.025207 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.025096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:35:51.077156 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.077109 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng"] Jul 28 17:35:51.085910 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.085881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:51.087614 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.087495 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng"] Jul 28 17:35:51.091214 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.088334 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 17:35:51.091214 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.088615 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9vwgj\"" Jul 28 17:35:51.181473 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.181433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c37bb557-15c5-4322-98cb-333737867adf-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2hcng\" (UID: \"c37bb557-15c5-4322-98cb-333737867adf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:51.181669 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.181551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rrl\" (UniqueName: \"kubernetes.io/projected/c37bb557-15c5-4322-98cb-333737867adf-kube-api-access-q6rrl\") pod \"monitoring-plugin-68b7fc9574-2hcng\" (UID: \"c37bb557-15c5-4322-98cb-333737867adf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:51.188462 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.187669 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cb98457bb-tnm4k"] Jul 28 17:35:51.191623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.191591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.194304 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.194278 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 17:35:51.194771 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.194739 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 17:35:51.194927 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.194908 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 17:35:51.194999 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.194974 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 17:35:51.194999 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.194982 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zr967\"" Jul 28 17:35:51.195339 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.195315 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 17:35:51.200780 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.200754 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb98457bb-tnm4k"] Jul 28 17:35:51.282365 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.282274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s69h8\" (UniqueName: \"kubernetes.io/projected/22e12b18-c342-4cef-b499-758d1f59b33d-kube-api-access-s69h8\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.282365 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.282332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rrl\" (UniqueName: \"kubernetes.io/projected/c37bb557-15c5-4322-98cb-333737867adf-kube-api-access-q6rrl\") pod \"monitoring-plugin-68b7fc9574-2hcng\" (UID: \"c37bb557-15c5-4322-98cb-333737867adf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:51.282365 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.282364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-oauth-config\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.282621 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.282402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-service-ca\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.282621 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.282443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-serving-cert\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.282621 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.282478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-oauth-serving-cert\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.282621 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.282561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c37bb557-15c5-4322-98cb-333737867adf-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2hcng\" (UID: \"c37bb557-15c5-4322-98cb-333737867adf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:51.282798 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.282626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-console-config\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.285627 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.285600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c37bb557-15c5-4322-98cb-333737867adf-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2hcng\" (UID: \"c37bb557-15c5-4322-98cb-333737867adf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:51.292710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.292657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rrl\" (UniqueName: \"kubernetes.io/projected/c37bb557-15c5-4322-98cb-333737867adf-kube-api-access-q6rrl\") pod \"monitoring-plugin-68b7fc9574-2hcng\" (UID: \"c37bb557-15c5-4322-98cb-333737867adf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:51.384096 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.384034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-service-ca\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.384908 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.384104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-serving-cert\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.384908 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.384370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-oauth-serving-cert\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.384908 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.384454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-console-config\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.384908 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.384508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s69h8\" (UniqueName: \"kubernetes.io/projected/22e12b18-c342-4cef-b499-758d1f59b33d-kube-api-access-s69h8\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.384908 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.384543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-oauth-config\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.384908 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.384845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-service-ca\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.385397 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.385107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-console-config\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.385397 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.385275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-oauth-serving-cert\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.387504 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.387482 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-oauth-config\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.387614 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.387529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-serving-cert\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.394385 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.394357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s69h8\" (UniqueName: \"kubernetes.io/projected/22e12b18-c342-4cef-b499-758d1f59b33d-kube-api-access-s69h8\") pod \"console-cb98457bb-tnm4k\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.401935 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.401904 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:51.504475 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.504442 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:35:51.672759 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.672727 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f947b7b5b-qdrhs"] Jul 28 17:35:51.678767 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:51.678731 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42744d32_298e_42d4_9cc7_5b4b38f1dbf2.slice/crio-055007283a60f6770e759f9a69eec6c9ea4cca6eb51e9883b2409f976ef481a5 WatchSource:0}: Error finding container 055007283a60f6770e759f9a69eec6c9ea4cca6eb51e9883b2409f976ef481a5: Status 404 returned error can't find the container with id 055007283a60f6770e759f9a69eec6c9ea4cca6eb51e9883b2409f976ef481a5 Jul 28 17:35:51.907973 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.907942 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng"] Jul 28 17:35:51.909167 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.909090 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb98457bb-tnm4k"] Jul 28 17:35:51.911264 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:51.911230 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37bb557_15c5_4322_98cb_333737867adf.slice/crio-08e0cd7b1458bc7abdefb4b63470b28c03b0b47340e7492c12e51fa5bcad4624 WatchSource:0}: Error finding container 08e0cd7b1458bc7abdefb4b63470b28c03b0b47340e7492c12e51fa5bcad4624: Status 404 returned error can't find the container with id 08e0cd7b1458bc7abdefb4b63470b28c03b0b47340e7492c12e51fa5bcad4624 Jul 28 17:35:51.911937 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:51.911900 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e12b18_c342_4cef_b499_758d1f59b33d.slice/crio-714bc39c91828e2d5f67c9064101c3390af28dabdc7c537a6b64bf22b82f5835 WatchSource:0}: Error finding container 714bc39c91828e2d5f67c9064101c3390af28dabdc7c537a6b64bf22b82f5835: Status 404 returned error can't find the container with id 714bc39c91828e2d5f67c9064101c3390af28dabdc7c537a6b64bf22b82f5835 Jul 28 17:35:51.968652 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.968579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb98457bb-tnm4k" event={"ID":"22e12b18-c342-4cef-b499-758d1f59b33d","Type":"ContainerStarted","Data":"714bc39c91828e2d5f67c9064101c3390af28dabdc7c537a6b64bf22b82f5835"} Jul 28 17:35:51.970232 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.970205 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" event={"ID":"c37bb557-15c5-4322-98cb-333737867adf","Type":"ContainerStarted","Data":"08e0cd7b1458bc7abdefb4b63470b28c03b0b47340e7492c12e51fa5bcad4624"} Jul 28 17:35:51.972863 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.972832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" event={"ID":"c093872e-6524-4834-9b1f-1e2dbd258945","Type":"ContainerStarted","Data":"6b90a5248795a5de6359ac42ed1d01dc4ee4febd4dcf272ef31eb94caf0d919c"} Jul 28 17:35:51.972994 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.972905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" event={"ID":"c093872e-6524-4834-9b1f-1e2dbd258945","Type":"ContainerStarted","Data":"ed57d2912f84c0e58faaefbbd6c5d62329711733a5b4d4ba5810b6ac8a9a5703"} Jul 28 17:35:51.972994 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.972950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" event={"ID":"c093872e-6524-4834-9b1f-1e2dbd258945","Type":"ContainerStarted","Data":"5678513ecafd437b9e8886d9695bd3ace3b985337a99b73b38fd22b0f64ddc9b"} Jul 28 17:35:51.974354 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:51.974330 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" event={"ID":"42744d32-298e-42d4-9cc7-5b4b38f1dbf2","Type":"ContainerStarted","Data":"055007283a60f6770e759f9a69eec6c9ea4cca6eb51e9883b2409f976ef481a5"} Jul 28 17:35:52.698591 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.698550 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:35:52.703110 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.703083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.706448 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.705439 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 17:35:52.706448 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.705442 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 17:35:52.706448 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.705667 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9rfjt\"" Jul 28 17:35:52.706448 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.705905 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 17:35:52.706448 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.705913 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3f22no18qkaq4\"" Jul 28 17:35:52.706448 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.706314 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 17:35:52.706448 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.706324 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 17:35:52.708379 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.706562 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 17:35:52.708379 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.706571 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 17:35:52.708379 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.706694 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 17:35:52.708379 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.707010 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 17:35:52.708379 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.707750 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 17:35:52.708379 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.708164 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 17:35:52.712061 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.711985 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 17:35:52.720804 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.720781 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 17:35:52.721693 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.721670 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 17:35:52.725632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.725610 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-config\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmz69\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-kube-api-access-cmz69\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.802223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.803147 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.803147 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.803147 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.803147 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.801911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-config\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.904129 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.905167 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.905167 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.905167 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.903978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmz69\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-kube-api-access-cmz69\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.905167 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.904952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.905167 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.905652 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.905167 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.906056 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.905167 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.906416 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.905167 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.906614 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.907959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.908330 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.908401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.909610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.913420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.914048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.914114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.915569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.915980 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-config\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.916734 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.917768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.917915 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.920286 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.918133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.921145 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.921118 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:52.923550 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:52.923490 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmz69\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-kube-api-access-cmz69\") pod \"prometheus-k8s-0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:53.027432 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:53.027346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:35:54.541806 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:54.541711 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:35:54.546949 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:35:54.546878 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd549fbd_5330_4360_8fda_09e60bfbeba0.slice/crio-968d6e2e0973e2e4729d4b53abaee39fdd810741ee6dfe07be9c73f83a145909 WatchSource:0}: Error finding container 968d6e2e0973e2e4729d4b53abaee39fdd810741ee6dfe07be9c73f83a145909: Status 404 returned error can't find the container with id 968d6e2e0973e2e4729d4b53abaee39fdd810741ee6dfe07be9c73f83a145909 Jul 28 17:35:54.996724 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:54.996430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerStarted","Data":"3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c"} Jul 28 17:35:54.996724 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:54.996522 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerStarted","Data":"d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246"} Jul 28 17:35:54.996724 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:54.996561 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerStarted","Data":"b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15"} Jul 28 17:35:54.996724 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:54.996597 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerStarted","Data":"36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f"} Jul 28 17:35:54.996724 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:54.996637 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerStarted","Data":"ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a"} Jul 28 17:35:54.996724 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:54.996674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerStarted","Data":"c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a"} Jul 28 17:35:55.006973 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.006923 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" event={"ID":"c093872e-6524-4834-9b1f-1e2dbd258945","Type":"ContainerStarted","Data":"5ff876e505ec920e856b216189aa0a6340014fd199df26d24bcb7f5184fb1c43"} Jul 28 17:35:55.007194 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.007065 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" event={"ID":"c093872e-6524-4834-9b1f-1e2dbd258945","Type":"ContainerStarted","Data":"d85fb7cfc370c31f5d537b5e26202625b050f3c77f9ebbb1dfdd05b1b7d94078"} Jul 28 17:35:55.007194 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.007128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" event={"ID":"c093872e-6524-4834-9b1f-1e2dbd258945","Type":"ContainerStarted","Data":"384f57b2ccf1eac4fb4be90a41b13ea023555506eb3198ec70ac08c76f1203cb"} Jul 28 17:35:55.007334 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.007207 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:35:55.009916 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.009876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" event={"ID":"42744d32-298e-42d4-9cc7-5b4b38f1dbf2","Type":"ContainerStarted","Data":"71f39a4ff45a52602b5063a409a15aaf420883b17ee480f86fc71ac38b08974c"} Jul 28 17:35:55.012012 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.011959 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" event={"ID":"c37bb557-15c5-4322-98cb-333737867adf","Type":"ContainerStarted","Data":"627467835c03a8aeb402bd4e32fac969598330475135ed3516e8e6a0e12a0981"} Jul 28 17:35:55.012145 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.012134 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:55.014342 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.014317 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerID="b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630" exitCode=0 Jul 28 17:35:55.014450 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.014360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerDied","Data":"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630"} Jul 28 17:35:55.014450 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.014438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerStarted","Data":"968d6e2e0973e2e4729d4b53abaee39fdd810741ee6dfe07be9c73f83a145909"} Jul 28 17:35:55.018760 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.018742 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" Jul 28 17:35:55.035374 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.035161 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.1393301510000002 podStartE2EDuration="8.035140471s" totalImagesPullingTime="4.868921199s" totalInitContainerRuntime="26.889121ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:55.034351271 +0000 UTC m=+80.006858822" watchObservedRunningTime="2026-07-28 17:35:55.035140471 +0000 UTC m=+80.007648024" Jul 28 17:35:55.052738 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.052677 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hcng" podStartSLOduration=1.594149424 podStartE2EDuration="4.052650807s" totalImagesPullingTime="2.458501383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:55.052603441 +0000 UTC m=+80.025110992" watchObservedRunningTime="2026-07-28 17:35:55.052650807 +0000 UTC m=+80.025158359" Jul 28 17:35:55.075773 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.075581 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" podStartSLOduration=2.390871848 podStartE2EDuration="5.075562762s" totalImagesPullingTime="2.684690914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:55.073445898 +0000 UTC m=+80.045953495" watchObservedRunningTime="2026-07-28 17:35:55.075562762 +0000 UTC m=+80.048070313" Jul 28 17:35:55.127038 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:55.126887 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" podStartSLOduration=2.221582303 podStartE2EDuration="7.126865581s" totalImagesPullingTime="4.905283278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:55.124898944 +0000 UTC m=+80.097406491" watchObservedRunningTime="2026-07-28 17:35:55.126865581 +0000 UTC m=+80.099373135" Jul 28 17:35:56.025428 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:35:56.025398 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67f7f7b7b8-fq4ws" Jul 28 17:36:04.365236 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:04.365092 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb98457bb-tnm4k"] Jul 28 17:36:06.073709 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:06.073665 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb98457bb-tnm4k" event={"ID":"22e12b18-c342-4cef-b499-758d1f59b33d","Type":"ContainerStarted","Data":"70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8"} Jul 28 17:36:06.076816 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:06.076764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerStarted","Data":"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e"} Jul 28 17:36:06.076945 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:06.076847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerStarted","Data":"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724"} Jul 28 17:36:06.080150 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:06.079986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-49w5r" event={"ID":"7bf7b156-f793-493b-9f1a-91afef4fb73f","Type":"ContainerStarted","Data":"16f2715b6ed3035217a1bad0dd77f928c1749c38a8f23185fee2d69d8a70615f"} Jul 28 17:36:06.080741 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:06.080671 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-49w5r" Jul 28 17:36:06.093298 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:06.093268 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-49w5r" Jul 28 17:36:06.093457 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:06.093307 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cb98457bb-tnm4k" podStartSLOduration=1.387188247 podStartE2EDuration="15.093292243s" totalImagesPullingTime="13.706103996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:06.091658648 +0000 UTC m=+91.064166196" watchObservedRunningTime="2026-07-28 17:36:06.093292243 +0000 UTC m=+91.065799794" Jul 28 17:36:06.115195 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:06.112959 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-49w5r" podStartSLOduration=1.362284483 podStartE2EDuration="24.112938438s" totalImagesPullingTime="22.750653955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:06.112646196 +0000 UTC m=+91.085153748" watchObservedRunningTime="2026-07-28 17:36:06.112938438 +0000 UTC m=+91.085445969" Jul 28 17:36:07.090525 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:07.090474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerStarted","Data":"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c"} Jul 28 17:36:07.091109 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:07.090611 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerStarted","Data":"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9"} Jul 28 17:36:07.091109 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:07.090663 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerStarted","Data":"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590"} Jul 28 17:36:07.091109 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:07.090706 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerStarted","Data":"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13"} Jul 28 17:36:07.125042 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:07.124977 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.258319509 podStartE2EDuration="15.124945046s" totalImagesPullingTime="10.84706264s" totalInitContainerRuntime="19.562897ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:07.122967035 +0000 UTC m=+92.095474611" watchObservedRunningTime="2026-07-28 17:36:07.124945046 +0000 UTC m=+92.097452631" Jul 28 17:36:08.027952 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:08.027915 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:36:11.026322 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:11.026268 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:36:11.026756 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:11.026346 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:36:11.505301 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:11.505265 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:36:18.926966 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:18.926937 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mq7v6" Jul 28 17:36:31.031299 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.031268 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:36:31.035329 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.035301 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f947b7b5b-qdrhs" Jul 28 17:36:31.100687 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.100643 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cb98457bb-tnm4k" podUID="22e12b18-c342-4cef-b499-758d1f59b33d" containerName="console" containerID="cri-o://70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8" gracePeriod=15 Jul 28 17:36:31.376926 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.376891 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:36:31.478025 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.477979 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s69h8\" (UniqueName: \"kubernetes.io/projected/22e12b18-c342-4cef-b499-758d1f59b33d-kube-api-access-s69h8\") pod \"22e12b18-c342-4cef-b499-758d1f59b33d\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " Jul 28 17:36:31.478250 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.478083 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-serving-cert\") pod \"22e12b18-c342-4cef-b499-758d1f59b33d\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " Jul 28 17:36:31.478250 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.478111 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-console-config\") pod \"22e12b18-c342-4cef-b499-758d1f59b33d\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " Jul 28 17:36:31.478250 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.478139 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-oauth-config\") pod \"22e12b18-c342-4cef-b499-758d1f59b33d\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " Jul 28 17:36:31.478250 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.478162 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-oauth-serving-cert\") pod \"22e12b18-c342-4cef-b499-758d1f59b33d\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " Jul 28 17:36:31.478250 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.478240 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-service-ca\") pod \"22e12b18-c342-4cef-b499-758d1f59b33d\" (UID: \"22e12b18-c342-4cef-b499-758d1f59b33d\") " Jul 28 17:36:31.478641 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.478614 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "22e12b18-c342-4cef-b499-758d1f59b33d" (UID: "22e12b18-c342-4cef-b499-758d1f59b33d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:36:31.478641 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.478624 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-console-config" (OuterVolumeSpecName: "console-config") pod "22e12b18-c342-4cef-b499-758d1f59b33d" (UID: "22e12b18-c342-4cef-b499-758d1f59b33d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:36:31.478762 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.478679 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-service-ca" (OuterVolumeSpecName: "service-ca") pod "22e12b18-c342-4cef-b499-758d1f59b33d" (UID: "22e12b18-c342-4cef-b499-758d1f59b33d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:36:31.480416 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.480388 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "22e12b18-c342-4cef-b499-758d1f59b33d" (UID: "22e12b18-c342-4cef-b499-758d1f59b33d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:36:31.480526 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.480433 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "22e12b18-c342-4cef-b499-758d1f59b33d" (UID: "22e12b18-c342-4cef-b499-758d1f59b33d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:36:31.480526 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.480515 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e12b18-c342-4cef-b499-758d1f59b33d-kube-api-access-s69h8" (OuterVolumeSpecName: "kube-api-access-s69h8") pod "22e12b18-c342-4cef-b499-758d1f59b33d" (UID: "22e12b18-c342-4cef-b499-758d1f59b33d"). InnerVolumeSpecName "kube-api-access-s69h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:36:31.579754 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.579660 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-serving-cert\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:36:31.579754 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.579694 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-console-config\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:36:31.579754 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.579703 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22e12b18-c342-4cef-b499-758d1f59b33d-console-oauth-config\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:36:31.579754 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.579712 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-oauth-serving-cert\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:36:31.579754 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.579721 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22e12b18-c342-4cef-b499-758d1f59b33d-service-ca\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:36:31.579754 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:31.579730 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s69h8\" (UniqueName: \"kubernetes.io/projected/22e12b18-c342-4cef-b499-758d1f59b33d-kube-api-access-s69h8\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:36:32.202556 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:32.202523 2576 generic.go:356] "Generic (PLEG): container finished" podID="22e12b18-c342-4cef-b499-758d1f59b33d" containerID="70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8" exitCode=0 Jul 28 17:36:32.203102 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:32.202568 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb98457bb-tnm4k" Jul 28 17:36:32.203102 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:32.202610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb98457bb-tnm4k" event={"ID":"22e12b18-c342-4cef-b499-758d1f59b33d","Type":"ContainerDied","Data":"70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8"} Jul 28 17:36:32.203102 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:32.202668 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb98457bb-tnm4k" event={"ID":"22e12b18-c342-4cef-b499-758d1f59b33d","Type":"ContainerDied","Data":"714bc39c91828e2d5f67c9064101c3390af28dabdc7c537a6b64bf22b82f5835"} Jul 28 17:36:32.203102 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:32.202699 2576 scope.go:117] "RemoveContainer" containerID="70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8" Jul 28 17:36:32.220866 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:32.220848 2576 scope.go:117] "RemoveContainer" containerID="70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8" Jul 28 17:36:32.221217 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:36:32.221193 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8\": container with ID starting with 70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8 not found: ID does not exist" containerID="70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8" Jul 28 17:36:32.221316 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:32.221221 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8"} err="failed to get container status \"70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8\": rpc error: code = NotFound desc = could not find container \"70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8\": container with ID starting with 70dc1d15f4d1e461635c95bfddd0fa8143e8bbab1c2314a583b31fe4ddfdd8e8 not found: ID does not exist" Jul 28 17:36:32.224831 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:32.224802 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb98457bb-tnm4k"] Jul 28 17:36:32.230335 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:32.230312 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cb98457bb-tnm4k"] Jul 28 17:36:33.594886 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:33.594854 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e12b18-c342-4cef-b499-758d1f59b33d" path="/var/lib/kubelet/pods/22e12b18-c342-4cef-b499-758d1f59b33d/volumes" Jul 28 17:36:53.028604 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:53.028567 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:36:53.091814 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:53.091788 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:36:53.308091 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:36:53.308022 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:06.698793 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:06.698760 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:37:06.699342 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:06.699009 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="alertmanager" containerID="cri-o://c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a" gracePeriod=120 Jul 28 17:37:06.699342 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:06.699073 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy-metric" containerID="cri-o://d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246" gracePeriod=120 Jul 28 17:37:06.699342 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:06.699096 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy-web" containerID="cri-o://36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f" gracePeriod=120 Jul 28 17:37:06.699342 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:06.699108 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="config-reloader" containerID="cri-o://ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a" gracePeriod=120 Jul 28 17:37:06.699342 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:06.699188 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="prom-label-proxy" containerID="cri-o://3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c" gracePeriod=120 Jul 28 17:37:06.699342 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:06.699212 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy" containerID="cri-o://b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15" gracePeriod=120 Jul 28 17:37:07.358798 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:07.358766 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerID="3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c" exitCode=0 Jul 28 17:37:07.358798 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:07.358793 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerID="b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15" exitCode=0 Jul 28 17:37:07.358798 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:07.358799 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerID="ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a" exitCode=0 Jul 28 17:37:07.358798 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:07.358804 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerID="c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a" exitCode=0 Jul 28 17:37:07.359063 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:07.358844 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerDied","Data":"3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c"} Jul 28 17:37:07.359063 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:07.358963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerDied","Data":"b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15"} Jul 28 17:37:07.359063 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:07.359008 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerDied","Data":"ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a"} Jul 28 17:37:07.359063 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:07.359052 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerDied","Data":"c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a"} Jul 28 17:37:07.951142 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:07.951121 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.107964 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.107870 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-web\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.107964 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.107909 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-cluster-tls-config\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.107964 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.107934 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-volume\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.107964 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.107962 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbbc8\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-kube-api-access-zbbc8\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108010 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-trusted-ca-bundle\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108041 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-web-config\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108063 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-out\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108088 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108121 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108158 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-tls-assets\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108246 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-main-tls\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108314 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108289 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-metrics-client-ca\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108672 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108317 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-main-db\") pod \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\" (UID: \"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a\") " Jul 28 17:37:08.108672 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108460 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:08.108672 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108607 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.109020 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.108981 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:08.109799 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.109767 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:37:08.110712 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.110664 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-volume" (OuterVolumeSpecName: "config-volume") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:08.111373 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.111348 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:08.111518 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.111427 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:08.111609 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.111532 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:08.111882 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.111854 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-kube-api-access-zbbc8" (OuterVolumeSpecName: "kube-api-access-zbbc8") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "kube-api-access-zbbc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:37:08.112301 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.112281 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:37:08.112503 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.112484 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-out" (OuterVolumeSpecName: "config-out") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:37:08.112921 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.112897 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:08.115239 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.115166 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:08.121287 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.121265 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-web-config" (OuterVolumeSpecName: "web-config") pod "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" (UID: "0b709b3c-28c2-4a16-9f54-2b2a69c2d36a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:08.209325 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209286 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbbc8\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-kube-api-access-zbbc8\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209325 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209318 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-web-config\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209325 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209327 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-out\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209325 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209337 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209589 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209347 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209589 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209356 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-tls-assets\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209589 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209365 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-main-tls\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209589 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209373 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-metrics-client-ca\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209589 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209382 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-alertmanager-main-db\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209589 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209391 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209589 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209401 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-cluster-tls-config\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.209589 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.209409 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a-config-volume\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:08.365890 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.365810 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerID="d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246" exitCode=0 Jul 28 17:37:08.365890 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.365834 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerID="36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f" exitCode=0 Jul 28 17:37:08.366069 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.365896 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.366069 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.365896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerDied","Data":"d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246"} Jul 28 17:37:08.366069 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.365989 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerDied","Data":"36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f"} Jul 28 17:37:08.366069 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.366036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b709b3c-28c2-4a16-9f54-2b2a69c2d36a","Type":"ContainerDied","Data":"f79e682d5df917bc9ab8a13e2d6622db331ba79f48cecd11091090b9d7600378"} Jul 28 17:37:08.366069 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.366049 2576 scope.go:117] "RemoveContainer" containerID="3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c" Jul 28 17:37:08.373823 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.373693 2576 scope.go:117] "RemoveContainer" containerID="d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246" Jul 28 17:37:08.381149 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.381127 2576 scope.go:117] "RemoveContainer" containerID="b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15" Jul 28 17:37:08.388238 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.388219 2576 scope.go:117] "RemoveContainer" containerID="36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f" Jul 28 17:37:08.395427 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.395410 2576 scope.go:117] "RemoveContainer" containerID="ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a" Jul 28 17:37:08.395560 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.395538 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:37:08.399727 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.399707 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:37:08.402930 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.402916 2576 scope.go:117] "RemoveContainer" containerID="c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a" Jul 28 17:37:08.409822 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.409799 2576 scope.go:117] "RemoveContainer" containerID="4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4" Jul 28 17:37:08.417011 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.416991 2576 scope.go:117] "RemoveContainer" containerID="3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c" Jul 28 17:37:08.417341 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:08.417320 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c\": container with ID starting with 3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c not found: ID does not exist" containerID="3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c" Jul 28 17:37:08.417412 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.417357 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c"} err="failed to get container status \"3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c\": rpc error: code = NotFound desc = could not find container \"3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c\": container with ID starting with 3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c not found: ID does not exist" Jul 28 17:37:08.417412 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.417387 2576 scope.go:117] "RemoveContainer" containerID="d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246" Jul 28 17:37:08.417631 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:08.417615 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246\": container with ID starting with d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246 not found: ID does not exist" containerID="d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246" Jul 28 17:37:08.417672 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.417637 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246"} err="failed to get container status \"d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246\": rpc error: code = NotFound desc = could not find container \"d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246\": container with ID starting with d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246 not found: ID does not exist" Jul 28 17:37:08.417672 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.417653 2576 scope.go:117] "RemoveContainer" containerID="b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15" Jul 28 17:37:08.417869 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:08.417854 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15\": container with ID starting with b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15 not found: ID does not exist" containerID="b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15" Jul 28 17:37:08.417914 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.417873 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15"} err="failed to get container status \"b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15\": rpc error: code = NotFound desc = could not find container \"b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15\": container with ID starting with b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15 not found: ID does not exist" Jul 28 17:37:08.417914 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.417886 2576 scope.go:117] "RemoveContainer" containerID="36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f" Jul 28 17:37:08.418072 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:08.418059 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f\": container with ID starting with 36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f not found: ID does not exist" containerID="36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f" Jul 28 17:37:08.418109 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.418076 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f"} err="failed to get container status \"36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f\": rpc error: code = NotFound desc = could not find container \"36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f\": container with ID starting with 36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f not found: ID does not exist" Jul 28 17:37:08.418109 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.418088 2576 scope.go:117] "RemoveContainer" containerID="ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a" Jul 28 17:37:08.418406 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:08.418388 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a\": container with ID starting with ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a not found: ID does not exist" containerID="ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a" Jul 28 17:37:08.418489 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.418412 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a"} err="failed to get container status \"ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a\": rpc error: code = NotFound desc = could not find container \"ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a\": container with ID starting with ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a not found: ID does not exist" Jul 28 17:37:08.418489 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.418431 2576 scope.go:117] "RemoveContainer" containerID="c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a" Jul 28 17:37:08.418651 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:08.418636 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a\": container with ID starting with c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a not found: ID does not exist" containerID="c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a" Jul 28 17:37:08.418687 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.418655 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a"} err="failed to get container status \"c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a\": rpc error: code = NotFound desc = could not find container \"c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a\": container with ID starting with c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a not found: ID does not exist" Jul 28 17:37:08.418687 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.418667 2576 scope.go:117] "RemoveContainer" containerID="4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4" Jul 28 17:37:08.418845 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:08.418830 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4\": container with ID starting with 4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4 not found: ID does not exist" containerID="4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4" Jul 28 17:37:08.418887 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.418847 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4"} err="failed to get container status \"4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4\": rpc error: code = NotFound desc = could not find container \"4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4\": container with ID starting with 4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4 not found: ID does not exist" Jul 28 17:37:08.418887 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.418858 2576 scope.go:117] "RemoveContainer" containerID="3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c" Jul 28 17:37:08.419103 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419089 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c"} err="failed to get container status \"3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c\": rpc error: code = NotFound desc = could not find container \"3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c\": container with ID starting with 3693f07ae38fd09e7ab8ad1ffb87f459ff96aedc1a8ec18a23adc66733130c9c not found: ID does not exist" Jul 28 17:37:08.419157 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419104 2576 scope.go:117] "RemoveContainer" containerID="d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246" Jul 28 17:37:08.419328 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419308 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246"} err="failed to get container status \"d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246\": rpc error: code = NotFound desc = could not find container \"d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246\": container with ID starting with d8587b1fef8bb75147d697624e4dba22f43e77308e41d813220a6466f195f246 not found: ID does not exist" Jul 28 17:37:08.419368 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419330 2576 scope.go:117] "RemoveContainer" containerID="b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15" Jul 28 17:37:08.419537 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419522 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15"} err="failed to get container status \"b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15\": rpc error: code = NotFound desc = could not find container \"b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15\": container with ID starting with b5b8c4382536411cc8790076473f0b440d6e0d3734c9b6ebaa2ac5f42028ec15 not found: ID does not exist" Jul 28 17:37:08.419576 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419537 2576 scope.go:117] "RemoveContainer" containerID="36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f" Jul 28 17:37:08.419730 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419715 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f"} err="failed to get container status \"36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f\": rpc error: code = NotFound desc = could not find container \"36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f\": container with ID starting with 36b94ad81469a1638dcc08a7eefd62c046e5b0168089a60dba61ca7d3d65191f not found: ID does not exist" Jul 28 17:37:08.419776 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419730 2576 scope.go:117] "RemoveContainer" containerID="ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a" Jul 28 17:37:08.419916 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419899 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a"} err="failed to get container status \"ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a\": rpc error: code = NotFound desc = could not find container \"ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a\": container with ID starting with ae12bbce6d825ec01f4f2cc3057e3d1c32c1d013fb888d24e762c66ad63d293a not found: ID does not exist" Jul 28 17:37:08.419954 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.419917 2576 scope.go:117] "RemoveContainer" containerID="c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a" Jul 28 17:37:08.420106 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.420091 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a"} err="failed to get container status \"c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a\": rpc error: code = NotFound desc = could not find container \"c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a\": container with ID starting with c6ab01a3338d8f9b7e6967eff064b65fd4df027aa9279187d6a9c8870d49c70a not found: ID does not exist" Jul 28 17:37:08.420147 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.420106 2576 scope.go:117] "RemoveContainer" containerID="4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4" Jul 28 17:37:08.420312 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.420297 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4"} err="failed to get container status \"4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4\": rpc error: code = NotFound desc = could not find container \"4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4\": container with ID starting with 4f7bc117d467644f876d589f38f333b550eb8704b8c57e0897826f723cfdf5f4 not found: ID does not exist" Jul 28 17:37:08.433757 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.433737 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:37:08.434032 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434020 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy-web" Jul 28 17:37:08.434072 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434033 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy-web" Jul 28 17:37:08.434072 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434050 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e12b18-c342-4cef-b499-758d1f59b33d" containerName="console" Jul 28 17:37:08.434072 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434057 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e12b18-c342-4cef-b499-758d1f59b33d" containerName="console" Jul 28 17:37:08.434072 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434065 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="config-reloader" Jul 28 17:37:08.434072 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434070 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="config-reloader" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434078 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434083 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434089 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy-metric" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434095 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy-metric" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434103 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="alertmanager" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434108 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="alertmanager" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434114 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="prom-label-proxy" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434119 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="prom-label-proxy" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434131 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="init-config-reloader" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434136 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="init-config-reloader" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434195 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy-web" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434202 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="prom-label-proxy" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434210 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="config-reloader" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434218 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434226 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="kube-rbac-proxy-metric" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434231 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e12b18-c342-4cef-b499-758d1f59b33d" containerName="console" Jul 28 17:37:08.434231 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.434238 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" containerName="alertmanager" Jul 28 17:37:08.437551 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.437534 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.439876 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.439858 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 17:37:08.440008 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.439992 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 17:37:08.440055 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.440009 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 17:37:08.440127 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.440112 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 17:37:08.440201 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.440141 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 17:37:08.440201 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.440113 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 17:37:08.440201 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.440116 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 17:37:08.440347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.440149 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hw99m\"" Jul 28 17:37:08.440347 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.440278 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 17:37:08.446623 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.446449 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 17:37:08.458079 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.458028 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:37:08.511789 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.511753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.511789 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.511788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.511992 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.511811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twl45\" (UniqueName: \"kubernetes.io/projected/2d693787-0acb-4c04-9f64-ae5b88cdf424-kube-api-access-twl45\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.511992 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.511871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d693787-0acb-4c04-9f64-ae5b88cdf424-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.511992 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.511891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2d693787-0acb-4c04-9f64-ae5b88cdf424-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.511992 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.511910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-config-volume\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.511992 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.511927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d693787-0acb-4c04-9f64-ae5b88cdf424-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.511992 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.511949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.512202 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.511994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.512202 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.512021 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.512202 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.512050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d693787-0acb-4c04-9f64-ae5b88cdf424-config-out\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.512202 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.512068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d693787-0acb-4c04-9f64-ae5b88cdf424-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.512202 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.512103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-web-config\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613338 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-config-volume\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613338 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d693787-0acb-4c04-9f64-ae5b88cdf424-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613604 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613604 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613604 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613604 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d693787-0acb-4c04-9f64-ae5b88cdf424-config-out\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613793 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d693787-0acb-4c04-9f64-ae5b88cdf424-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613793 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-web-config\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613793 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613793 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613976 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twl45\" (UniqueName: \"kubernetes.io/projected/2d693787-0acb-4c04-9f64-ae5b88cdf424-kube-api-access-twl45\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613976 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d693787-0acb-4c04-9f64-ae5b88cdf424-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.613976 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.613898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2d693787-0acb-4c04-9f64-ae5b88cdf424-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.614146 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.614116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d693787-0acb-4c04-9f64-ae5b88cdf424-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.614289 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.614264 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2d693787-0acb-4c04-9f64-ae5b88cdf424-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.616679 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.616458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d693787-0acb-4c04-9f64-ae5b88cdf424-config-out\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.616679 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.616492 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.616679 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.616547 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.616679 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.616583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-config-volume\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.616679 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.616601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.617000 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.616784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d693787-0acb-4c04-9f64-ae5b88cdf424-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.617000 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.616815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.617166 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.617146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.617522 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.617504 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d693787-0acb-4c04-9f64-ae5b88cdf424-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.618352 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.618335 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d693787-0acb-4c04-9f64-ae5b88cdf424-web-config\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.622958 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.622941 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twl45\" (UniqueName: \"kubernetes.io/projected/2d693787-0acb-4c04-9f64-ae5b88cdf424-kube-api-access-twl45\") pod \"alertmanager-main-0\" (UID: \"2d693787-0acb-4c04-9f64-ae5b88cdf424\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.749241 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.749204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:37:08.886957 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:08.886884 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:37:08.890041 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:37:08.890012 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d693787_0acb_4c04_9f64_ae5b88cdf424.slice/crio-c921545ded03910548eeb50e4b3af492bef913bc9ce0f6d00ea7156a835d42cd WatchSource:0}: Error finding container c921545ded03910548eeb50e4b3af492bef913bc9ce0f6d00ea7156a835d42cd: Status 404 returned error can't find the container with id c921545ded03910548eeb50e4b3af492bef913bc9ce0f6d00ea7156a835d42cd Jul 28 17:37:09.371946 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:09.371906 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d693787-0acb-4c04-9f64-ae5b88cdf424" containerID="ae02f4c496f750b8bcb16750728b587ad2de71dfc499a87587a560396b924092" exitCode=0 Jul 28 17:37:09.372463 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:09.371991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d693787-0acb-4c04-9f64-ae5b88cdf424","Type":"ContainerDied","Data":"ae02f4c496f750b8bcb16750728b587ad2de71dfc499a87587a560396b924092"} Jul 28 17:37:09.372463 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:09.372079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d693787-0acb-4c04-9f64-ae5b88cdf424","Type":"ContainerStarted","Data":"c921545ded03910548eeb50e4b3af492bef913bc9ce0f6d00ea7156a835d42cd"} Jul 28 17:37:09.595757 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:09.595728 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b709b3c-28c2-4a16-9f54-2b2a69c2d36a" path="/var/lib/kubelet/pods/0b709b3c-28c2-4a16-9f54-2b2a69c2d36a/volumes" Jul 28 17:37:10.379915 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.379883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d693787-0acb-4c04-9f64-ae5b88cdf424","Type":"ContainerStarted","Data":"df56458e00903dce416c403c1463ae06610e7055779c2a91b394586def1c6c37"} Jul 28 17:37:10.380324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.379949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d693787-0acb-4c04-9f64-ae5b88cdf424","Type":"ContainerStarted","Data":"92507a2c8c2d1eda80e97bd707a30207eb1115351eea712d534e564c523b6256"} Jul 28 17:37:10.380324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.379978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d693787-0acb-4c04-9f64-ae5b88cdf424","Type":"ContainerStarted","Data":"dbc27dd084496e1fe2a91d157cfdfaa346b1cf4045f342fffb55f6c1ffd3df62"} Jul 28 17:37:10.380324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.380003 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d693787-0acb-4c04-9f64-ae5b88cdf424","Type":"ContainerStarted","Data":"84faf6a8d3615018f54066ffaf8e7b6b71815ee426b23a7ac3c07cf8f1638c9b"} Jul 28 17:37:10.380324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.380026 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d693787-0acb-4c04-9f64-ae5b88cdf424","Type":"ContainerStarted","Data":"e0a0b869dac4526aabe3e38cacaac4a6a86665f81a4eac2662bff2a794c1f766"} Jul 28 17:37:10.380324 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.380050 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d693787-0acb-4c04-9f64-ae5b88cdf424","Type":"ContainerStarted","Data":"7ee9ec13287856694a621015d80431c69ee2d74785a68a8b45eb9813c5a343db"} Jul 28 17:37:10.408814 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.408761 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.388115222 podStartE2EDuration="2.408732807s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.617585ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:10.407959842 +0000 UTC m=+155.380467392" watchObservedRunningTime="2026-07-28 17:37:10.408732807 +0000 UTC m=+155.381240582" Jul 28 17:37:10.723999 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.723962 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bd555bcf8-jdj52"] Jul 28 17:37:10.727229 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.727206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.730305 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.729514 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 17:37:10.730305 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.729545 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 17:37:10.730305 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.729743 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8h9qk\"" Jul 28 17:37:10.730305 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.729803 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 17:37:10.730305 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.729980 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 17:37:10.731433 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.731254 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 17:37:10.735449 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.735425 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 17:37:10.743055 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.743026 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bd555bcf8-jdj52"] Jul 28 17:37:10.834067 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.834023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj448\" (UniqueName: \"kubernetes.io/projected/c996fa72-bf21-4f6a-b4e0-89543925ea6c-kube-api-access-gj448\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.834285 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.834096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.834285 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.834126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c996fa72-bf21-4f6a-b4e0-89543925ea6c-metrics-client-ca\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.834285 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.834152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c996fa72-bf21-4f6a-b4e0-89543925ea6c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.834285 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.834196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-secret-telemeter-client\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.834285 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.834229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-federate-client-tls\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.834544 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.834326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c996fa72-bf21-4f6a-b4e0-89543925ea6c-serving-certs-ca-bundle\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.834544 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.834377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-telemeter-client-tls\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.934847 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.934812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c996fa72-bf21-4f6a-b4e0-89543925ea6c-serving-certs-ca-bundle\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.935002 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.934867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-telemeter-client-tls\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.935002 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.934898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj448\" (UniqueName: \"kubernetes.io/projected/c996fa72-bf21-4f6a-b4e0-89543925ea6c-kube-api-access-gj448\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.935002 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.934931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.935002 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.934955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c996fa72-bf21-4f6a-b4e0-89543925ea6c-metrics-client-ca\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.935002 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.934981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c996fa72-bf21-4f6a-b4e0-89543925ea6c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.935344 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.935006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-secret-telemeter-client\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.935344 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.935039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-federate-client-tls\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.935872 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.935818 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c996fa72-bf21-4f6a-b4e0-89543925ea6c-metrics-client-ca\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.935998 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.935868 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c996fa72-bf21-4f6a-b4e0-89543925ea6c-serving-certs-ca-bundle\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.936388 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.936362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c996fa72-bf21-4f6a-b4e0-89543925ea6c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.938327 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.938153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-federate-client-tls\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.938327 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.938228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-telemeter-client-tls\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.938327 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.938262 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-secret-telemeter-client\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.938963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.938925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c996fa72-bf21-4f6a-b4e0-89543925ea6c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:10.943066 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:10.943047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj448\" (UniqueName: \"kubernetes.io/projected/c996fa72-bf21-4f6a-b4e0-89543925ea6c-kube-api-access-gj448\") pod \"telemeter-client-5bd555bcf8-jdj52\" (UID: \"c996fa72-bf21-4f6a-b4e0-89543925ea6c\") " pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:11.040935 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.040842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" Jul 28 17:37:11.051099 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.051066 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:37:11.051422 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.051390 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="prometheus" containerID="cri-o://66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" gracePeriod=600 Jul 28 17:37:11.051506 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.051442 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" gracePeriod=600 Jul 28 17:37:11.051506 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.051468 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy-web" containerID="cri-o://38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" gracePeriod=600 Jul 28 17:37:11.051660 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.051568 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="config-reloader" containerID="cri-o://a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" gracePeriod=600 Jul 28 17:37:11.051660 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.051441 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="thanos-sidecar" containerID="cri-o://dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" gracePeriod=600 Jul 28 17:37:11.051785 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.051411 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy" containerID="cri-o://bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" gracePeriod=600 Jul 28 17:37:11.213040 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.213009 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bd555bcf8-jdj52"] Jul 28 17:37:11.216465 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:37:11.216437 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc996fa72_bf21_4f6a_b4e0_89543925ea6c.slice/crio-268d795f0105bec36333e79e745005b5125cab26714e45cdd37795a7c0e64756 WatchSource:0}: Error finding container 268d795f0105bec36333e79e745005b5125cab26714e45cdd37795a7c0e64756: Status 404 returned error can't find the container with id 268d795f0105bec36333e79e745005b5125cab26714e45cdd37795a7c0e64756 Jul 28 17:37:11.326311 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.326279 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.396066 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396030 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerID="17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" exitCode=0 Jul 28 17:37:11.396066 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396061 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerID="bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" exitCode=0 Jul 28 17:37:11.396066 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396070 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerID="38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" exitCode=0 Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396077 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerID="dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" exitCode=0 Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396084 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerID="a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" exitCode=0 Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396092 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerID="66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" exitCode=0 Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396122 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerDied","Data":"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c"} Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396290 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerDied","Data":"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9"} Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396334 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerDied","Data":"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590"} Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396374 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerDied","Data":"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13"} Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396402 2576 scope.go:117] "RemoveContainer" containerID="17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerDied","Data":"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e"} Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerDied","Data":"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724"} Jul 28 17:37:11.396632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.396490 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd549fbd-5330-4360-8fda-09e60bfbeba0","Type":"ContainerDied","Data":"968d6e2e0973e2e4729d4b53abaee39fdd810741ee6dfe07be9c73f83a145909"} Jul 28 17:37:11.397972 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.397942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" event={"ID":"c996fa72-bf21-4f6a-b4e0-89543925ea6c","Type":"ContainerStarted","Data":"268d795f0105bec36333e79e745005b5125cab26714e45cdd37795a7c0e64756"} Jul 28 17:37:11.407441 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.407418 2576 scope.go:117] "RemoveContainer" containerID="bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" Jul 28 17:37:11.417608 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.417586 2576 scope.go:117] "RemoveContainer" containerID="38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" Jul 28 17:37:11.424671 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.424653 2576 scope.go:117] "RemoveContainer" containerID="dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" Jul 28 17:37:11.431381 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.431364 2576 scope.go:117] "RemoveContainer" containerID="a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" Jul 28 17:37:11.438205 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.438187 2576 scope.go:117] "RemoveContainer" containerID="66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" Jul 28 17:37:11.440353 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440334 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmz69\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-kube-api-access-cmz69\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440437 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440369 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-1\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440437 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440415 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-tls\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440547 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440436 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-db\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440547 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440462 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-grpc-tls\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440676 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440568 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-tls-assets\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440735 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440715 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-kubelet-serving-ca-bundle\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440789 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440762 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-metrics-client-certs\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440837 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440798 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-kube-rbac-proxy\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440837 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440814 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:11.440939 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440849 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-0\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440939 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440891 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-2\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.440939 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440918 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-trusted-ca-bundle\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.441081 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-metrics-client-ca\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.441081 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.440980 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.441081 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441021 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-thanos-prometheus-http-client-file\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.441081 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441071 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-config-out\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.441301 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441124 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-web-config\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.441301 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441155 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.441301 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441214 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-config\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.441301 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441240 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-serving-certs-ca-bundle\") pod \"fd549fbd-5330-4360-8fda-09e60bfbeba0\" (UID: \"fd549fbd-5330-4360-8fda-09e60bfbeba0\") " Jul 28 17:37:11.441582 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441302 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:11.441582 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441384 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:11.441730 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441707 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.441788 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441735 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:11.441788 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441741 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.441788 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.441780 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.442775 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.442750 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:11.443502 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.443216 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:11.443502 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.443331 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:37:11.443693 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.443616 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:11.443693 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.443649 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:11.443796 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.443689 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-kube-api-access-cmz69" (OuterVolumeSpecName: "kube-api-access-cmz69") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "kube-api-access-cmz69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:37:11.444994 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.444955 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:37:11.444994 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.444956 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:11.445235 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.445195 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:11.445583 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.445549 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-config-out" (OuterVolumeSpecName: "config-out") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:37:11.445678 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.445634 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:11.445781 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.445756 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:11.445872 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.445849 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:11.447014 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.446993 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-config" (OuterVolumeSpecName: "config") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:11.447419 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.447398 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:11.448533 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.448511 2576 scope.go:117] "RemoveContainer" containerID="b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630" Jul 28 17:37:11.455385 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.455366 2576 scope.go:117] "RemoveContainer" containerID="17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" Jul 28 17:37:11.455647 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:11.455628 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": container with ID starting with 17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c not found: ID does not exist" containerID="17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" Jul 28 17:37:11.455721 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.455653 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c"} err="failed to get container status \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": rpc error: code = NotFound desc = could not find container \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": container with ID starting with 17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c not found: ID does not exist" Jul 28 17:37:11.455721 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.455669 2576 scope.go:117] "RemoveContainer" containerID="bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" Jul 28 17:37:11.455896 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:11.455880 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": container with ID starting with bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9 not found: ID does not exist" containerID="bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" Jul 28 17:37:11.455956 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.455898 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9"} err="failed to get container status \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": rpc error: code = NotFound desc = could not find container \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": container with ID starting with bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9 not found: ID does not exist" Jul 28 17:37:11.455956 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.455910 2576 scope.go:117] "RemoveContainer" containerID="38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" Jul 28 17:37:11.456109 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:11.456090 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": container with ID starting with 38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590 not found: ID does not exist" containerID="38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" Jul 28 17:37:11.456165 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.456113 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590"} err="failed to get container status \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": rpc error: code = NotFound desc = could not find container \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": container with ID starting with 38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590 not found: ID does not exist" Jul 28 17:37:11.456165 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.456131 2576 scope.go:117] "RemoveContainer" containerID="dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" Jul 28 17:37:11.456420 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:11.456398 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": container with ID starting with dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13 not found: ID does not exist" containerID="dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" Jul 28 17:37:11.456480 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.456422 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13"} err="failed to get container status \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": rpc error: code = NotFound desc = could not find container \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": container with ID starting with dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13 not found: ID does not exist" Jul 28 17:37:11.456480 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.456436 2576 scope.go:117] "RemoveContainer" containerID="a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" Jul 28 17:37:11.456672 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:11.456656 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": container with ID starting with a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e not found: ID does not exist" containerID="a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" Jul 28 17:37:11.456717 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.456677 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e"} err="failed to get container status \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": rpc error: code = NotFound desc = could not find container \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": container with ID starting with a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e not found: ID does not exist" Jul 28 17:37:11.456717 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.456697 2576 scope.go:117] "RemoveContainer" containerID="66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" Jul 28 17:37:11.456787 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.456723 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-web-config" (OuterVolumeSpecName: "web-config") pod "fd549fbd-5330-4360-8fda-09e60bfbeba0" (UID: "fd549fbd-5330-4360-8fda-09e60bfbeba0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:37:11.456925 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:11.456912 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": container with ID starting with 66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724 not found: ID does not exist" containerID="66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" Jul 28 17:37:11.456957 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.456927 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724"} err="failed to get container status \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": rpc error: code = NotFound desc = could not find container \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": container with ID starting with 66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724 not found: ID does not exist" Jul 28 17:37:11.456957 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.456939 2576 scope.go:117] "RemoveContainer" containerID="b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630" Jul 28 17:37:11.457141 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:37:11.457126 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": container with ID starting with b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630 not found: ID does not exist" containerID="b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630" Jul 28 17:37:11.457195 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457147 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630"} err="failed to get container status \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": rpc error: code = NotFound desc = could not find container \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": container with ID starting with b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630 not found: ID does not exist" Jul 28 17:37:11.457195 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457162 2576 scope.go:117] "RemoveContainer" containerID="17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" Jul 28 17:37:11.457379 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457364 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c"} err="failed to get container status \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": rpc error: code = NotFound desc = could not find container \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": container with ID starting with 17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c not found: ID does not exist" Jul 28 17:37:11.457413 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457380 2576 scope.go:117] "RemoveContainer" containerID="bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" Jul 28 17:37:11.457582 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457568 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9"} err="failed to get container status \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": rpc error: code = NotFound desc = could not find container \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": container with ID starting with bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9 not found: ID does not exist" Jul 28 17:37:11.457618 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457582 2576 scope.go:117] "RemoveContainer" containerID="38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" Jul 28 17:37:11.457754 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457739 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590"} err="failed to get container status \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": rpc error: code = NotFound desc = could not find container \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": container with ID starting with 38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590 not found: ID does not exist" Jul 28 17:37:11.457790 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457755 2576 scope.go:117] "RemoveContainer" containerID="dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" Jul 28 17:37:11.457940 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457921 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13"} err="failed to get container status \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": rpc error: code = NotFound desc = could not find container \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": container with ID starting with dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13 not found: ID does not exist" Jul 28 17:37:11.457973 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.457941 2576 scope.go:117] "RemoveContainer" containerID="a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" Jul 28 17:37:11.458137 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458121 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e"} err="failed to get container status \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": rpc error: code = NotFound desc = could not find container \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": container with ID starting with a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e not found: ID does not exist" Jul 28 17:37:11.458189 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458139 2576 scope.go:117] "RemoveContainer" containerID="66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" Jul 28 17:37:11.458373 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458358 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724"} err="failed to get container status \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": rpc error: code = NotFound desc = could not find container \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": container with ID starting with 66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724 not found: ID does not exist" Jul 28 17:37:11.458406 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458373 2576 scope.go:117] "RemoveContainer" containerID="b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630" Jul 28 17:37:11.458554 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458540 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630"} err="failed to get container status \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": rpc error: code = NotFound desc = could not find container \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": container with ID starting with b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630 not found: ID does not exist" Jul 28 17:37:11.458588 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458555 2576 scope.go:117] "RemoveContainer" containerID="17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" Jul 28 17:37:11.458783 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458763 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c"} err="failed to get container status \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": rpc error: code = NotFound desc = could not find container \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": container with ID starting with 17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c not found: ID does not exist" Jul 28 17:37:11.458832 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458783 2576 scope.go:117] "RemoveContainer" containerID="bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" Jul 28 17:37:11.458990 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458970 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9"} err="failed to get container status \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": rpc error: code = NotFound desc = could not find container \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": container with ID starting with bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9 not found: ID does not exist" Jul 28 17:37:11.459050 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.458992 2576 scope.go:117] "RemoveContainer" containerID="38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" Jul 28 17:37:11.459255 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.459236 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590"} err="failed to get container status \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": rpc error: code = NotFound desc = could not find container \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": container with ID starting with 38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590 not found: ID does not exist" Jul 28 17:37:11.459300 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.459257 2576 scope.go:117] "RemoveContainer" containerID="dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" Jul 28 17:37:11.459467 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.459450 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13"} err="failed to get container status \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": rpc error: code = NotFound desc = could not find container \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": container with ID starting with dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13 not found: ID does not exist" Jul 28 17:37:11.459535 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.459470 2576 scope.go:117] "RemoveContainer" containerID="a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" Jul 28 17:37:11.459702 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.459686 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e"} err="failed to get container status \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": rpc error: code = NotFound desc = could not find container \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": container with ID starting with a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e not found: ID does not exist" Jul 28 17:37:11.459746 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.459703 2576 scope.go:117] "RemoveContainer" containerID="66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" Jul 28 17:37:11.459939 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.459917 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724"} err="failed to get container status \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": rpc error: code = NotFound desc = could not find container \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": container with ID starting with 66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724 not found: ID does not exist" Jul 28 17:37:11.459994 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.459940 2576 scope.go:117] "RemoveContainer" containerID="b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630" Jul 28 17:37:11.460143 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.460127 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630"} err="failed to get container status \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": rpc error: code = NotFound desc = could not find container \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": container with ID starting with b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630 not found: ID does not exist" Jul 28 17:37:11.460243 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.460143 2576 scope.go:117] "RemoveContainer" containerID="17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" Jul 28 17:37:11.460395 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.460380 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c"} err="failed to get container status \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": rpc error: code = NotFound desc = could not find container \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": container with ID starting with 17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c not found: ID does not exist" Jul 28 17:37:11.460433 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.460395 2576 scope.go:117] "RemoveContainer" containerID="bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" Jul 28 17:37:11.460590 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.460574 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9"} err="failed to get container status \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": rpc error: code = NotFound desc = could not find container \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": container with ID starting with bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9 not found: ID does not exist" Jul 28 17:37:11.460627 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.460591 2576 scope.go:117] "RemoveContainer" containerID="38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" Jul 28 17:37:11.460807 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.460788 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590"} err="failed to get container status \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": rpc error: code = NotFound desc = could not find container \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": container with ID starting with 38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590 not found: ID does not exist" Jul 28 17:37:11.460807 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.460807 2576 scope.go:117] "RemoveContainer" containerID="dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" Jul 28 17:37:11.461036 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461023 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13"} err="failed to get container status \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": rpc error: code = NotFound desc = could not find container \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": container with ID starting with dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13 not found: ID does not exist" Jul 28 17:37:11.461071 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461036 2576 scope.go:117] "RemoveContainer" containerID="a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" Jul 28 17:37:11.461250 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461222 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e"} err="failed to get container status \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": rpc error: code = NotFound desc = could not find container \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": container with ID starting with a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e not found: ID does not exist" Jul 28 17:37:11.461250 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461246 2576 scope.go:117] "RemoveContainer" containerID="66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" Jul 28 17:37:11.461463 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461448 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724"} err="failed to get container status \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": rpc error: code = NotFound desc = could not find container \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": container with ID starting with 66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724 not found: ID does not exist" Jul 28 17:37:11.461463 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461462 2576 scope.go:117] "RemoveContainer" containerID="b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630" Jul 28 17:37:11.461636 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461622 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630"} err="failed to get container status \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": rpc error: code = NotFound desc = could not find container \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": container with ID starting with b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630 not found: ID does not exist" Jul 28 17:37:11.461677 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461636 2576 scope.go:117] "RemoveContainer" containerID="17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" Jul 28 17:37:11.461822 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461807 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c"} err="failed to get container status \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": rpc error: code = NotFound desc = could not find container \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": container with ID starting with 17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c not found: ID does not exist" Jul 28 17:37:11.461869 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461821 2576 scope.go:117] "RemoveContainer" containerID="bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" Jul 28 17:37:11.462004 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.461987 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9"} err="failed to get container status \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": rpc error: code = NotFound desc = could not find container \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": container with ID starting with bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9 not found: ID does not exist" Jul 28 17:37:11.462044 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462004 2576 scope.go:117] "RemoveContainer" containerID="38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" Jul 28 17:37:11.462165 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462151 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590"} err="failed to get container status \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": rpc error: code = NotFound desc = could not find container \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": container with ID starting with 38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590 not found: ID does not exist" Jul 28 17:37:11.462225 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462165 2576 scope.go:117] "RemoveContainer" containerID="dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" Jul 28 17:37:11.462405 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462388 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13"} err="failed to get container status \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": rpc error: code = NotFound desc = could not find container \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": container with ID starting with dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13 not found: ID does not exist" Jul 28 17:37:11.462474 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462407 2576 scope.go:117] "RemoveContainer" containerID="a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" Jul 28 17:37:11.462604 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462589 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e"} err="failed to get container status \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": rpc error: code = NotFound desc = could not find container \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": container with ID starting with a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e not found: ID does not exist" Jul 28 17:37:11.462646 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462604 2576 scope.go:117] "RemoveContainer" containerID="66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" Jul 28 17:37:11.462759 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462743 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724"} err="failed to get container status \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": rpc error: code = NotFound desc = could not find container \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": container with ID starting with 66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724 not found: ID does not exist" Jul 28 17:37:11.462801 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462759 2576 scope.go:117] "RemoveContainer" containerID="b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630" Jul 28 17:37:11.462940 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462927 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630"} err="failed to get container status \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": rpc error: code = NotFound desc = could not find container \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": container with ID starting with b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630 not found: ID does not exist" Jul 28 17:37:11.462982 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.462940 2576 scope.go:117] "RemoveContainer" containerID="17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c" Jul 28 17:37:11.463144 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463128 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c"} err="failed to get container status \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": rpc error: code = NotFound desc = could not find container \"17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c\": container with ID starting with 17fdf3da99ed853503a8ac8839226a7a2e38006dc7cd9d8162f5d597432df87c not found: ID does not exist" Jul 28 17:37:11.463144 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463143 2576 scope.go:117] "RemoveContainer" containerID="bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9" Jul 28 17:37:11.463363 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463348 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9"} err="failed to get container status \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": rpc error: code = NotFound desc = could not find container \"bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9\": container with ID starting with bb2f1d8ae558b3cf73b1ecea27184934637795a7eb6a7ea8dc5ddc8e082b57f9 not found: ID does not exist" Jul 28 17:37:11.463418 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463362 2576 scope.go:117] "RemoveContainer" containerID="38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590" Jul 28 17:37:11.463581 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463561 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590"} err="failed to get container status \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": rpc error: code = NotFound desc = could not find container \"38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590\": container with ID starting with 38e98e5f5275f56f8bcb6d760cf983d455153fda11eb648479a43ab642cbd590 not found: ID does not exist" Jul 28 17:37:11.463619 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463582 2576 scope.go:117] "RemoveContainer" containerID="dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13" Jul 28 17:37:11.463786 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463771 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13"} err="failed to get container status \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": rpc error: code = NotFound desc = could not find container \"dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13\": container with ID starting with dd558fbd79e0e78b55c05c8bcf3de77d5cf320b2f71250b7d8875aa2d596ce13 not found: ID does not exist" Jul 28 17:37:11.463831 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463785 2576 scope.go:117] "RemoveContainer" containerID="a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e" Jul 28 17:37:11.463950 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463935 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e"} err="failed to get container status \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": rpc error: code = NotFound desc = could not find container \"a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e\": container with ID starting with a0bc2eb94f729d2bea9bcde60b1913a0248b22ef21c7f5692b9561b4ef8c470e not found: ID does not exist" Jul 28 17:37:11.463998 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.463949 2576 scope.go:117] "RemoveContainer" containerID="66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724" Jul 28 17:37:11.464118 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.464103 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724"} err="failed to get container status \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": rpc error: code = NotFound desc = could not find container \"66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724\": container with ID starting with 66055bbc2a6660e1bc3980df8b3ea6081566b648bfba791aa5616c66b5c3d724 not found: ID does not exist" Jul 28 17:37:11.464157 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.464118 2576 scope.go:117] "RemoveContainer" containerID="b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630" Jul 28 17:37:11.464310 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.464294 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630"} err="failed to get container status \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": rpc error: code = NotFound desc = could not find container \"b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630\": container with ID starting with b6c64a992b33392e8b88656b4ef36afde61dc0fcbb4dd033f49c59755c4a2630 not found: ID does not exist" Jul 28 17:37:11.542614 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542545 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542614 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542580 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-db\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542614 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542596 2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-grpc-tls\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542614 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542608 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-tls-assets\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542614 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542620 2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-metrics-client-certs\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542614 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542633 2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-kube-rbac-proxy\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542648 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542660 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542674 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-metrics-client-ca\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542686 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542698 2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542710 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd549fbd-5330-4360-8fda-09e60bfbeba0-config-out\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542721 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-web-config\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542734 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542745 2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd549fbd-5330-4360-8fda-09e60bfbeba0-config\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542757 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd549fbd-5330-4360-8fda-09e60bfbeba0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.542963 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.542770 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmz69\" (UniqueName: \"kubernetes.io/projected/fd549fbd-5330-4360-8fda-09e60bfbeba0-kube-api-access-cmz69\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:37:11.719137 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.719106 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:37:11.725472 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.725446 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:37:11.753119 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753085 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:37:11.753543 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753515 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy-web" Jul 28 17:37:11.753543 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753541 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy-web" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753554 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy-thanos" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753563 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy-thanos" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753579 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="thanos-sidecar" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753588 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="thanos-sidecar" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753603 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="init-config-reloader" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753613 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="init-config-reloader" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753656 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="prometheus" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753665 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="prometheus" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753688 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="config-reloader" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753696 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="config-reloader" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753703 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy" Jul 28 17:37:11.753706 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753710 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy" Jul 28 17:37:11.754073 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753769 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy-thanos" Jul 28 17:37:11.754073 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753779 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="config-reloader" Jul 28 17:37:11.754073 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753786 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy" Jul 28 17:37:11.754073 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753791 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="kube-rbac-proxy-web" Jul 28 17:37:11.754073 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753798 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="prometheus" Jul 28 17:37:11.754073 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.753804 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" containerName="thanos-sidecar" Jul 28 17:37:11.756737 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.756710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.759708 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.759685 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 17:37:11.759814 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.759694 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 17:37:11.759875 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.759834 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 17:37:11.760116 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760015 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 17:37:11.760116 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760031 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 17:37:11.760116 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760020 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 17:37:11.760319 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760229 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3f22no18qkaq4\"" Jul 28 17:37:11.760319 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760306 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 17:37:11.760421 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760324 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 17:37:11.760421 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760329 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 17:37:11.760516 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760485 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 17:37:11.760725 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760670 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9rfjt\"" Jul 28 17:37:11.760831 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760721 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 17:37:11.760983 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.760965 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 17:37:11.764339 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.764307 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 17:37:11.766273 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.766256 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 17:37:11.778063 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.778034 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:37:11.845473 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs45c\" (UniqueName: \"kubernetes.io/projected/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-kube-api-access-rs45c\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845916 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845916 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845916 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845916 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.845916 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.846100 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.846100 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.845997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.846100 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.846014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.846100 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.846033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.846100 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.846055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.846100 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.846074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.846100 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.846092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-config\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.846432 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.846114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.946764 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.946734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.946983 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.946781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.946983 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.946816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.946983 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.946841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.946983 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.946864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.946983 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.946888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-config\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947239 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947239 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs45c\" (UniqueName: \"kubernetes.io/projected/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-kube-api-access-rs45c\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947239 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947239 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947408 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947408 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947408 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947408 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947408 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947627 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947627 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947627 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947627 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947627 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.947795 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.947734 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.948074 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.948047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.948221 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.948205 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.949320 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.949293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.949540 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.949514 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.950080 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.950053 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.952952 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.951187 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.952952 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.951453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.952952 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.951731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.952952 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.951934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.952952 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.952077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.952952 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.952544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.952952 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.952621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.952952 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.952852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.953451 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.953138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-config\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.953689 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.953659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.954059 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.954039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.954576 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.954551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.955019 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.955004 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:11.957348 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:11.957327 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs45c\" (UniqueName: \"kubernetes.io/projected/0fb08721-54bb-4ad0-aa2e-ba36ebe731a0-kube-api-access-rs45c\") pod \"prometheus-k8s-0\" (UID: \"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:12.072346 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:12.072311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:37:12.235072 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:12.234990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:37:12.238452 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:37:12.238422 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb08721_54bb_4ad0_aa2e_ba36ebe731a0.slice/crio-8ddebb71fbf0061b1e39ccc1d62d300ef23e037a6ef6e977783a521da9dfe5d6 WatchSource:0}: Error finding container 8ddebb71fbf0061b1e39ccc1d62d300ef23e037a6ef6e977783a521da9dfe5d6: Status 404 returned error can't find the container with id 8ddebb71fbf0061b1e39ccc1d62d300ef23e037a6ef6e977783a521da9dfe5d6 Jul 28 17:37:12.407701 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:12.407665 2576 generic.go:356] "Generic (PLEG): container finished" podID="0fb08721-54bb-4ad0-aa2e-ba36ebe731a0" containerID="846789a17451af1ae1ed5f57be89b8d64ce1384ff3308cd3c16018900a22b17e" exitCode=0 Jul 28 17:37:12.408155 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:12.407756 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0","Type":"ContainerDied","Data":"846789a17451af1ae1ed5f57be89b8d64ce1384ff3308cd3c16018900a22b17e"} Jul 28 17:37:12.408155 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:12.407864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0","Type":"ContainerStarted","Data":"8ddebb71fbf0061b1e39ccc1d62d300ef23e037a6ef6e977783a521da9dfe5d6"} Jul 28 17:37:13.416415 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.416378 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0","Type":"ContainerStarted","Data":"cb521e9f9c6534cbd69269cc5b4679652c31bd1e54c40e103acb1df3445c9a6e"} Jul 28 17:37:13.416813 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.416474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0","Type":"ContainerStarted","Data":"409a87f68c746df284b3773189fae9fcb39cf605fa15c94d4f295dcc24150a5e"} Jul 28 17:37:13.416813 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.416511 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0","Type":"ContainerStarted","Data":"dc51d990a9b24fd14a3c4fcb6e40a4c63567507c91c3b5b8215035abf6d19588"} Jul 28 17:37:13.416813 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.416538 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0","Type":"ContainerStarted","Data":"0b23c8324d9311d7308ecb27d320ad7d05c492c602f0b88746b03be40bd73c32"} Jul 28 17:37:13.416813 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.416573 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0","Type":"ContainerStarted","Data":"eea1d43e984d7106dba2012cabbd5c5ce38ad7d20f5655de16ec3f79a01f22fa"} Jul 28 17:37:13.416813 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.416610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fb08721-54bb-4ad0-aa2e-ba36ebe731a0","Type":"ContainerStarted","Data":"fed55251692613ee743090b37d9598c7e21fdb550d493cf9a17d159343929784"} Jul 28 17:37:13.418492 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.418466 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" event={"ID":"c996fa72-bf21-4f6a-b4e0-89543925ea6c","Type":"ContainerStarted","Data":"5f170e37744fab5f95a2e5b7fe2a7e8ffbe0e3a82687e64d63feadccba346acd"} Jul 28 17:37:13.418612 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.418511 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" event={"ID":"c996fa72-bf21-4f6a-b4e0-89543925ea6c","Type":"ContainerStarted","Data":"65f9db4e1e93b308aadcf1203e2587e05b165b567a203f311713633c3d3d6fce"} Jul 28 17:37:13.418612 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.418534 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" event={"ID":"c996fa72-bf21-4f6a-b4e0-89543925ea6c","Type":"ContainerStarted","Data":"b0e3d5152f1d8ddf61b14c0fd2effb0587c21e5e82da960867d5c40ca089bb3b"} Jul 28 17:37:13.451934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.451840 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.430170107 podStartE2EDuration="2.451820345s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.650238ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:13.449584069 +0000 UTC m=+158.422091620" watchObservedRunningTime="2026-07-28 17:37:13.451820345 +0000 UTC m=+158.424327923" Jul 28 17:37:13.470773 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.470714 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bd555bcf8-jdj52" podStartSLOduration=1.7210211960000001 podStartE2EDuration="3.470692304s" totalImagesPullingTime="1.749671108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:13.469002534 +0000 UTC m=+158.441510085" watchObservedRunningTime="2026-07-28 17:37:13.470692304 +0000 UTC m=+158.443199854" Jul 28 17:37:13.596829 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:13.596798 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd549fbd-5330-4360-8fda-09e60bfbeba0" path="/var/lib/kubelet/pods/fd549fbd-5330-4360-8fda-09e60bfbeba0/volumes" Jul 28 17:37:17.072910 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:37:17.072867 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:38:12.073335 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:38:12.073297 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:38:12.089030 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:38:12.089004 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:38:12.674041 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:38:12.674016 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:39:35.400426 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:39:35.400388 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 17:40:05.967729 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:05.967694 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-prwlw"] Jul 28 17:40:05.970696 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:05.970682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:05.972927 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:05.972907 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 17:40:05.973082 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:05.973057 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 17:40:05.973423 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:05.973408 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2zpfc\"" Jul 28 17:40:05.981778 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:05.981424 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-prwlw"] Jul 28 17:40:06.052306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:06.052263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjlj\" (UniqueName: \"kubernetes.io/projected/73ef2638-ed48-4870-a0ce-c719ff8a8401-kube-api-access-kbjlj\") pod \"cert-manager-webhook-6486ffd997-prwlw\" (UID: \"73ef2638-ed48-4870-a0ce-c719ff8a8401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:06.052306 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:06.052308 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ef2638-ed48-4870-a0ce-c719ff8a8401-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-prwlw\" (UID: \"73ef2638-ed48-4870-a0ce-c719ff8a8401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:06.152865 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:06.152823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjlj\" (UniqueName: \"kubernetes.io/projected/73ef2638-ed48-4870-a0ce-c719ff8a8401-kube-api-access-kbjlj\") pod \"cert-manager-webhook-6486ffd997-prwlw\" (UID: \"73ef2638-ed48-4870-a0ce-c719ff8a8401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:06.152865 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:06.152868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ef2638-ed48-4870-a0ce-c719ff8a8401-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-prwlw\" (UID: \"73ef2638-ed48-4870-a0ce-c719ff8a8401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:06.161223 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:06.161191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ef2638-ed48-4870-a0ce-c719ff8a8401-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-prwlw\" (UID: \"73ef2638-ed48-4870-a0ce-c719ff8a8401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:06.161361 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:06.161345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjlj\" (UniqueName: \"kubernetes.io/projected/73ef2638-ed48-4870-a0ce-c719ff8a8401-kube-api-access-kbjlj\") pod \"cert-manager-webhook-6486ffd997-prwlw\" (UID: \"73ef2638-ed48-4870-a0ce-c719ff8a8401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:06.289071 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:06.288991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:06.423568 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:06.423542 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-prwlw"] Jul 28 17:40:06.425976 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:40:06.425950 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73ef2638_ed48_4870_a0ce_c719ff8a8401.slice/crio-9f3984da33acb11c20bc799426b9629dccd0a758ca60bd5459fc77db54d7c02c WatchSource:0}: Error finding container 9f3984da33acb11c20bc799426b9629dccd0a758ca60bd5459fc77db54d7c02c: Status 404 returned error can't find the container with id 9f3984da33acb11c20bc799426b9629dccd0a758ca60bd5459fc77db54d7c02c Jul 28 17:40:06.427855 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:06.427836 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:40:07.119561 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:07.119531 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" event={"ID":"73ef2638-ed48-4870-a0ce-c719ff8a8401","Type":"ContainerStarted","Data":"9f3984da33acb11c20bc799426b9629dccd0a758ca60bd5459fc77db54d7c02c"} Jul 28 17:40:10.136207 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:10.136105 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" event={"ID":"73ef2638-ed48-4870-a0ce-c719ff8a8401","Type":"ContainerStarted","Data":"1b063891496a8c1975b5e4f9e1f5d73a661e9700c806a7661bfa0b395c5537d5"} Jul 28 17:40:10.136577 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:10.136237 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:10.154695 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:10.154642 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" podStartSLOduration=1.826379598 podStartE2EDuration="5.154623506s" totalImagesPullingTime="3.328243908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:10.153679636 +0000 UTC m=+335.126187185" watchObservedRunningTime="2026-07-28 17:40:10.154623506 +0000 UTC m=+335.127131055" Jul 28 17:40:14.878083 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:14.878007 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qb864"] Jul 28 17:40:14.881394 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:14.881376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qb864" Jul 28 17:40:14.883421 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:14.883400 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-g456v\"" Jul 28 17:40:14.890954 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:14.890926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qb864"] Jul 28 17:40:14.928261 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:14.928214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27617dd1-9e38-41b9-b284-dc70df1671a1-bound-sa-token\") pod \"cert-manager-55b4d954d5-qb864\" (UID: \"27617dd1-9e38-41b9-b284-dc70df1671a1\") " pod="cert-manager/cert-manager-55b4d954d5-qb864" Jul 28 17:40:14.928457 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:14.928292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48fj\" (UniqueName: \"kubernetes.io/projected/27617dd1-9e38-41b9-b284-dc70df1671a1-kube-api-access-h48fj\") pod \"cert-manager-55b4d954d5-qb864\" (UID: \"27617dd1-9e38-41b9-b284-dc70df1671a1\") " pod="cert-manager/cert-manager-55b4d954d5-qb864" Jul 28 17:40:15.029531 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:15.029493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27617dd1-9e38-41b9-b284-dc70df1671a1-bound-sa-token\") pod \"cert-manager-55b4d954d5-qb864\" (UID: \"27617dd1-9e38-41b9-b284-dc70df1671a1\") " pod="cert-manager/cert-manager-55b4d954d5-qb864" Jul 28 17:40:15.029736 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:15.029544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48fj\" (UniqueName: \"kubernetes.io/projected/27617dd1-9e38-41b9-b284-dc70df1671a1-kube-api-access-h48fj\") pod \"cert-manager-55b4d954d5-qb864\" (UID: \"27617dd1-9e38-41b9-b284-dc70df1671a1\") " pod="cert-manager/cert-manager-55b4d954d5-qb864" Jul 28 17:40:15.038943 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:15.038912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27617dd1-9e38-41b9-b284-dc70df1671a1-bound-sa-token\") pod \"cert-manager-55b4d954d5-qb864\" (UID: \"27617dd1-9e38-41b9-b284-dc70df1671a1\") " pod="cert-manager/cert-manager-55b4d954d5-qb864" Jul 28 17:40:15.039243 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:15.039068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48fj\" (UniqueName: \"kubernetes.io/projected/27617dd1-9e38-41b9-b284-dc70df1671a1-kube-api-access-h48fj\") pod \"cert-manager-55b4d954d5-qb864\" (UID: \"27617dd1-9e38-41b9-b284-dc70df1671a1\") " pod="cert-manager/cert-manager-55b4d954d5-qb864" Jul 28 17:40:15.190770 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:15.190734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qb864" Jul 28 17:40:15.320061 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:15.320037 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qb864"] Jul 28 17:40:15.321772 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:40:15.321746 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27617dd1_9e38_41b9_b284_dc70df1671a1.slice/crio-ec9aca8bbcfc44c127c65f4eaeadf113e45bd460dfdb99ccc1089b163866aac2 WatchSource:0}: Error finding container ec9aca8bbcfc44c127c65f4eaeadf113e45bd460dfdb99ccc1089b163866aac2: Status 404 returned error can't find the container with id ec9aca8bbcfc44c127c65f4eaeadf113e45bd460dfdb99ccc1089b163866aac2 Jul 28 17:40:16.143403 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:16.143377 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-prwlw" Jul 28 17:40:16.162904 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:16.162871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qb864" event={"ID":"27617dd1-9e38-41b9-b284-dc70df1671a1","Type":"ContainerStarted","Data":"26ebcd811f6963eec430f4a1dfb4ba47f57de05dd76ef9f686a8cc99386276cd"} Jul 28 17:40:16.163076 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:16.162914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qb864" event={"ID":"27617dd1-9e38-41b9-b284-dc70df1671a1","Type":"ContainerStarted","Data":"ec9aca8bbcfc44c127c65f4eaeadf113e45bd460dfdb99ccc1089b163866aac2"} Jul 28 17:40:16.181632 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:16.181579 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-qb864" podStartSLOduration=2.181561542 podStartE2EDuration="2.181561542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:16.180996625 +0000 UTC m=+341.153504168" watchObservedRunningTime="2026-07-28 17:40:16.181561542 +0000 UTC m=+341.154069092" Jul 28 17:40:37.555113 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.555038 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l"] Jul 28 17:40:37.559207 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.559169 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.561320 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.561300 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 17:40:37.561629 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.561614 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 17:40:37.561770 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.561752 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 17:40:37.561826 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.561781 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-djnth\"" Jul 28 17:40:37.561870 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.561838 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 17:40:37.596319 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.596289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l"] Jul 28 17:40:37.727163 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.727131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-tmp\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.727339 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.727168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.727339 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.727268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.727339 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.727307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zkw\" (UniqueName: \"kubernetes.io/projected/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-kube-api-access-99zkw\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.727441 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.727343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-manifests\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.828301 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.828223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-tmp\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.828301 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.828257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.828490 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.828352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.828490 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.828412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zkw\" (UniqueName: \"kubernetes.io/projected/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-kube-api-access-99zkw\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.828490 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.828453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-manifests\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.828826 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.828805 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-manifests\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.830636 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.830612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-tmp\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.830766 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.830723 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.830926 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.830907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.841308 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.841280 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zkw\" (UniqueName: \"kubernetes.io/projected/1d066b5a-7b39-4c48-b5c1-58bb3befbd85-kube-api-access-99zkw\") pod \"opendatahub-operator-controller-manager-5c7c5665b5-sc52l\" (UID: \"1d066b5a-7b39-4c48-b5c1-58bb3befbd85\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:37.869106 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:37.869076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:38.000660 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:38.000635 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l"] Jul 28 17:40:38.002804 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:40:38.002773 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d066b5a_7b39_4c48_b5c1_58bb3befbd85.slice/crio-1a461bcb0d9f9555e05d54c13acda30c60705b1b243bd7d196fb95bedbe18fee WatchSource:0}: Error finding container 1a461bcb0d9f9555e05d54c13acda30c60705b1b243bd7d196fb95bedbe18fee: Status 404 returned error can't find the container with id 1a461bcb0d9f9555e05d54c13acda30c60705b1b243bd7d196fb95bedbe18fee Jul 28 17:40:38.253296 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:38.253258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" event={"ID":"1d066b5a-7b39-4c48-b5c1-58bb3befbd85","Type":"ContainerStarted","Data":"1a461bcb0d9f9555e05d54c13acda30c60705b1b243bd7d196fb95bedbe18fee"} Jul 28 17:40:41.268069 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:41.268032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" event={"ID":"1d066b5a-7b39-4c48-b5c1-58bb3befbd85","Type":"ContainerStarted","Data":"3873b34f0c152e2197e49934049a2c0d1c04b6900a5cdd467e29ea58cfc03c23"} Jul 28 17:40:42.274233 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:42.274114 2576 generic.go:356] "Generic (PLEG): container finished" podID="1d066b5a-7b39-4c48-b5c1-58bb3befbd85" containerID="3873b34f0c152e2197e49934049a2c0d1c04b6900a5cdd467e29ea58cfc03c23" exitCode=0 Jul 28 17:40:42.274233 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:42.274200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" event={"ID":"1d066b5a-7b39-4c48-b5c1-58bb3befbd85","Type":"ContainerDied","Data":"3873b34f0c152e2197e49934049a2c0d1c04b6900a5cdd467e29ea58cfc03c23"} Jul 28 17:40:43.280745 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:43.280704 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" event={"ID":"1d066b5a-7b39-4c48-b5c1-58bb3befbd85","Type":"ContainerStarted","Data":"50dd344ad4db2a66aeb0f8fdb397c8b01d2df3d282a9b03571a3d5c0668037a4"} Jul 28 17:40:43.281249 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:43.280809 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:43.338185 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:43.338114 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" podStartSLOduration=2.069168067 podStartE2EDuration="6.338085315s" totalImagesPullingTime="3.242656691s" totalInitContainerRuntime="1.026260557s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:43.33764683 +0000 UTC m=+368.310154399" watchObservedRunningTime="2026-07-28 17:40:43.338085315 +0000 UTC m=+368.310592863" Jul 28 17:40:54.287446 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:54.287413 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c7c5665b5-sc52l" Jul 28 17:40:55.599471 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.599444 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr"] Jul 28 17:40:55.607501 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.607476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.610438 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.610417 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:40:55.610578 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.610488 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 17:40:55.610664 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.610417 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 17:40:55.610664 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.610418 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 17:40:55.610770 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.610754 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-prcts\"" Jul 28 17:40:55.610864 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.610845 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 17:40:55.612687 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.612665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr"] Jul 28 17:40:55.685717 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.685685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8909975-4b7a-44aa-959e-7d3376e735ea-cert\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.685717 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.685721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbsm\" (UniqueName: \"kubernetes.io/projected/f8909975-4b7a-44aa-959e-7d3376e735ea-kube-api-access-rhbsm\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.685922 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.685798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8909975-4b7a-44aa-959e-7d3376e735ea-manager-config\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.685922 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.685829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8909975-4b7a-44aa-959e-7d3376e735ea-metrics-cert\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.787096 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.787062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8909975-4b7a-44aa-959e-7d3376e735ea-manager-config\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.787281 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.787105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8909975-4b7a-44aa-959e-7d3376e735ea-metrics-cert\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.787281 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.787233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8909975-4b7a-44aa-959e-7d3376e735ea-cert\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.787281 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.787264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbsm\" (UniqueName: \"kubernetes.io/projected/f8909975-4b7a-44aa-959e-7d3376e735ea-kube-api-access-rhbsm\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.787825 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.787805 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8909975-4b7a-44aa-959e-7d3376e735ea-manager-config\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.789525 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.789505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8909975-4b7a-44aa-959e-7d3376e735ea-metrics-cert\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.789624 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.789528 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8909975-4b7a-44aa-959e-7d3376e735ea-cert\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.796545 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.796524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbsm\" (UniqueName: \"kubernetes.io/projected/f8909975-4b7a-44aa-959e-7d3376e735ea-kube-api-access-rhbsm\") pod \"lws-controller-manager-8c847c87f-gscjr\" (UID: \"f8909975-4b7a-44aa-959e-7d3376e735ea\") " pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:55.921654 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:55.921622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:56.048704 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:56.048681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr"] Jul 28 17:40:56.051245 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:40:56.051219 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8909975_4b7a_44aa_959e_7d3376e735ea.slice/crio-232c5a3069e7c9dd0ebac4bb4098a6d79d5d3f58686ce4126dc805d0df195421 WatchSource:0}: Error finding container 232c5a3069e7c9dd0ebac4bb4098a6d79d5d3f58686ce4126dc805d0df195421: Status 404 returned error can't find the container with id 232c5a3069e7c9dd0ebac4bb4098a6d79d5d3f58686ce4126dc805d0df195421 Jul 28 17:40:56.338053 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:56.337970 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" event={"ID":"f8909975-4b7a-44aa-959e-7d3376e735ea","Type":"ContainerStarted","Data":"232c5a3069e7c9dd0ebac4bb4098a6d79d5d3f58686ce4126dc805d0df195421"} Jul 28 17:40:59.353781 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:59.353741 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" event={"ID":"f8909975-4b7a-44aa-959e-7d3376e735ea","Type":"ContainerStarted","Data":"5cde4bfaeedcf75937484e17c2496fc928cd1987c845c2dd62fefbbe23833a82"} Jul 28 17:40:59.354210 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:59.353873 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:40:59.373369 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:40:59.373315 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" podStartSLOduration=1.883621569 podStartE2EDuration="4.373298526s" totalImagesPullingTime="2.489676957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:59.372347117 +0000 UTC m=+384.344854664" watchObservedRunningTime="2026-07-28 17:40:59.373298526 +0000 UTC m=+384.345806076" Jul 28 17:41:07.260374 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.260334 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc"] Jul 28 17:41:07.264350 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.264325 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:07.269300 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.269275 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 17:41:07.269436 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.269304 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 17:41:07.269858 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.269837 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xdnjx\"" Jul 28 17:41:07.270319 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.270303 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 28 17:41:07.270452 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.270437 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 28 17:41:07.292981 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.292939 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc"] Jul 28 17:41:07.397669 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.397634 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/287d6281-c6a8-4cdf-a0c9-187778f10002-tmp\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:07.397669 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.397683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhpt\" (UniqueName: \"kubernetes.io/projected/287d6281-c6a8-4cdf-a0c9-187778f10002-kube-api-access-bqhpt\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:07.397941 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.397749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/287d6281-c6a8-4cdf-a0c9-187778f10002-tls-certs\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:07.499219 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.499182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/287d6281-c6a8-4cdf-a0c9-187778f10002-tmp\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:07.499427 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.499225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhpt\" (UniqueName: \"kubernetes.io/projected/287d6281-c6a8-4cdf-a0c9-187778f10002-kube-api-access-bqhpt\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:07.499427 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.499267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/287d6281-c6a8-4cdf-a0c9-187778f10002-tls-certs\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:07.499552 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:41:07.499471 2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 28 17:41:07.499601 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:41:07.499563 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287d6281-c6a8-4cdf-a0c9-187778f10002-tls-certs podName:287d6281-c6a8-4cdf-a0c9-187778f10002 nodeName:}" failed. No retries permitted until 2026-07-28 17:41:07.999538867 +0000 UTC m=+392.972046411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/287d6281-c6a8-4cdf-a0c9-187778f10002-tls-certs") pod "kube-auth-proxy-6b8f9cc6c-ssgcc" (UID: "287d6281-c6a8-4cdf-a0c9-187778f10002") : secret "kube-auth-proxy-tls" not found Jul 28 17:41:07.501518 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.501487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/287d6281-c6a8-4cdf-a0c9-187778f10002-tmp\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:07.522330 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:07.522248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhpt\" (UniqueName: \"kubernetes.io/projected/287d6281-c6a8-4cdf-a0c9-187778f10002-kube-api-access-bqhpt\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:08.012869 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:08.012830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/287d6281-c6a8-4cdf-a0c9-187778f10002-tls-certs\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:08.015450 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:08.015422 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/287d6281-c6a8-4cdf-a0c9-187778f10002-tls-certs\") pod \"kube-auth-proxy-6b8f9cc6c-ssgcc\" (UID: \"287d6281-c6a8-4cdf-a0c9-187778f10002\") " pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:08.175598 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:08.175564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" Jul 28 17:41:08.317903 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:08.317869 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc"] Jul 28 17:41:08.320664 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:41:08.320637 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287d6281_c6a8_4cdf_a0c9_187778f10002.slice/crio-d12793a30fafa72857c5d721c0f94f8ab096fd450b16ca7bd4a325cccfe107fe WatchSource:0}: Error finding container d12793a30fafa72857c5d721c0f94f8ab096fd450b16ca7bd4a325cccfe107fe: Status 404 returned error can't find the container with id d12793a30fafa72857c5d721c0f94f8ab096fd450b16ca7bd4a325cccfe107fe Jul 28 17:41:08.395871 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:08.395838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" event={"ID":"287d6281-c6a8-4cdf-a0c9-187778f10002","Type":"ContainerStarted","Data":"d12793a30fafa72857c5d721c0f94f8ab096fd450b16ca7bd4a325cccfe107fe"} Jul 28 17:41:10.361470 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:10.361443 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8c847c87f-gscjr" Jul 28 17:41:11.415816 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:11.415783 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" event={"ID":"287d6281-c6a8-4cdf-a0c9-187778f10002","Type":"ContainerStarted","Data":"4af6077ba583c53ea92fc4c77b5fb7ec8a16f941ef8b4af139ed6cfef78fa079"} Jul 28 17:41:11.434998 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:41:11.434942 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b8f9cc6c-ssgcc" podStartSLOduration=1.422793315 podStartE2EDuration="4.434921969s" totalImagesPullingTime="3.012128654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:11.433899396 +0000 UTC m=+396.406406937" watchObservedRunningTime="2026-07-28 17:41:11.434921969 +0000 UTC m=+396.407429545" Jul 28 17:42:45.386998 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.386966 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk"] Jul 28 17:42:45.390289 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.390268 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:45.394018 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.393989 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 28 17:42:45.395293 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.395276 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 17:42:45.395849 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.395831 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 28 17:42:45.395961 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.395853 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x977p\"" Jul 28 17:42:45.395961 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.395867 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 17:42:45.412407 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.412374 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk"] Jul 28 17:42:45.468387 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.468354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98584ef1-bcb8-45ed-89ed-4397e2be4676-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:45.468545 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.468493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/98584ef1-bcb8-45ed-89ed-4397e2be4676-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:45.468545 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.468523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgxf\" (UniqueName: \"kubernetes.io/projected/98584ef1-bcb8-45ed-89ed-4397e2be4676-kube-api-access-ndgxf\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:45.568953 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.568924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/98584ef1-bcb8-45ed-89ed-4397e2be4676-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:45.569153 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.568958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgxf\" (UniqueName: \"kubernetes.io/projected/98584ef1-bcb8-45ed-89ed-4397e2be4676-kube-api-access-ndgxf\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:45.569153 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.569003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98584ef1-bcb8-45ed-89ed-4397e2be4676-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:45.569153 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:42:45.569103 2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 28 17:42:45.569341 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:42:45.569198 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98584ef1-bcb8-45ed-89ed-4397e2be4676-plugin-serving-cert podName:98584ef1-bcb8-45ed-89ed-4397e2be4676 nodeName:}" failed. No retries permitted until 2026-07-28 17:42:46.069152486 +0000 UTC m=+491.041660015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/98584ef1-bcb8-45ed-89ed-4397e2be4676-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-77lsk" (UID: "98584ef1-bcb8-45ed-89ed-4397e2be4676") : secret "plugin-serving-cert" not found Jul 28 17:42:45.569655 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.569633 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/98584ef1-bcb8-45ed-89ed-4397e2be4676-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:45.584043 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:45.584011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgxf\" (UniqueName: \"kubernetes.io/projected/98584ef1-bcb8-45ed-89ed-4397e2be4676-kube-api-access-ndgxf\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:46.073123 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:46.073092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98584ef1-bcb8-45ed-89ed-4397e2be4676-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:46.073300 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:42:46.073252 2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 28 17:42:46.073345 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:42:46.073320 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98584ef1-bcb8-45ed-89ed-4397e2be4676-plugin-serving-cert podName:98584ef1-bcb8-45ed-89ed-4397e2be4676 nodeName:}" failed. No retries permitted until 2026-07-28 17:42:47.07330305 +0000 UTC m=+492.045810590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/98584ef1-bcb8-45ed-89ed-4397e2be4676-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-77lsk" (UID: "98584ef1-bcb8-45ed-89ed-4397e2be4676") : secret "plugin-serving-cert" not found Jul 28 17:42:47.083145 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:47.083109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98584ef1-bcb8-45ed-89ed-4397e2be4676-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:47.085650 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:47.085624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98584ef1-bcb8-45ed-89ed-4397e2be4676-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-77lsk\" (UID: \"98584ef1-bcb8-45ed-89ed-4397e2be4676\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:47.199250 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:47.199211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" Jul 28 17:42:47.336595 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:47.336571 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk"] Jul 28 17:42:47.338646 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:42:47.338617 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98584ef1_bcb8_45ed_89ed_4397e2be4676.slice/crio-384602679eaf80bed7609ef85c782b1b254d02ce99ae3daf4f40decce762cdd7 WatchSource:0}: Error finding container 384602679eaf80bed7609ef85c782b1b254d02ce99ae3daf4f40decce762cdd7: Status 404 returned error can't find the container with id 384602679eaf80bed7609ef85c782b1b254d02ce99ae3daf4f40decce762cdd7 Jul 28 17:42:47.850508 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:47.850474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" event={"ID":"98584ef1-bcb8-45ed-89ed-4397e2be4676","Type":"ContainerStarted","Data":"384602679eaf80bed7609ef85c782b1b254d02ce99ae3daf4f40decce762cdd7"} Jul 28 17:42:55.098260 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.098219 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8"] Jul 28 17:42:55.105534 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.105498 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:42:55.107931 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.107877 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z9zdn\"" Jul 28 17:42:55.113055 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.113025 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8"] Jul 28 17:42:55.267377 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.267338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/675c979c-f640-4428-a73e-efe7f017cc95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" (UID: \"675c979c-f640-4428-a73e-efe7f017cc95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:42:55.267550 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.267487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z68d\" (UniqueName: \"kubernetes.io/projected/675c979c-f640-4428-a73e-efe7f017cc95-kube-api-access-6z68d\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" (UID: \"675c979c-f640-4428-a73e-efe7f017cc95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:42:55.368350 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.368270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/675c979c-f640-4428-a73e-efe7f017cc95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" (UID: \"675c979c-f640-4428-a73e-efe7f017cc95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:42:55.368561 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.368373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z68d\" (UniqueName: \"kubernetes.io/projected/675c979c-f640-4428-a73e-efe7f017cc95-kube-api-access-6z68d\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" (UID: \"675c979c-f640-4428-a73e-efe7f017cc95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:42:55.368689 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.368659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/675c979c-f640-4428-a73e-efe7f017cc95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" (UID: \"675c979c-f640-4428-a73e-efe7f017cc95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:42:55.385064 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.385034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z68d\" (UniqueName: \"kubernetes.io/projected/675c979c-f640-4428-a73e-efe7f017cc95-kube-api-access-6z68d\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" (UID: \"675c979c-f640-4428-a73e-efe7f017cc95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:42:55.420015 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.419978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:42:55.914639 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.914604 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8"] Jul 28 17:42:55.934543 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.934494 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8"] Jul 28 17:42:55.997788 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:55.997758 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k"] Jul 28 17:42:56.004482 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:56.004457 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:42:56.022311 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:56.021717 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k"] Jul 28 17:42:56.175983 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:56.175890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9g6\" (UniqueName: \"kubernetes.io/projected/24639594-3084-4a56-9e0c-6d18b0fa0bc3-kube-api-access-xf9g6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h6l8k\" (UID: \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:42:56.175983 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:56.175939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24639594-3084-4a56-9e0c-6d18b0fa0bc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h6l8k\" (UID: \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:42:56.276511 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:56.276478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9g6\" (UniqueName: \"kubernetes.io/projected/24639594-3084-4a56-9e0c-6d18b0fa0bc3-kube-api-access-xf9g6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h6l8k\" (UID: \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:42:56.276711 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:56.276537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24639594-3084-4a56-9e0c-6d18b0fa0bc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h6l8k\" (UID: \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:42:56.276917 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:56.276899 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24639594-3084-4a56-9e0c-6d18b0fa0bc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h6l8k\" (UID: \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:42:56.291512 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:56.291481 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9g6\" (UniqueName: \"kubernetes.io/projected/24639594-3084-4a56-9e0c-6d18b0fa0bc3-kube-api-access-xf9g6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h6l8k\" (UID: \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:42:56.319450 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:42:56.319416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:43:08.104137 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:08.104106 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k"] Jul 28 17:43:08.675487 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:08.675315 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k"] Jul 28 17:43:08.678280 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:43:08.678245 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24639594_3084_4a56_9e0c_6d18b0fa0bc3.slice/crio-70a6de75159d237782d15a0222e34eb0a12a4c6b7b9936646345a299f1fa4700 WatchSource:0}: Error finding container 70a6de75159d237782d15a0222e34eb0a12a4c6b7b9936646345a299f1fa4700: Status 404 returned error can't find the container with id 70a6de75159d237782d15a0222e34eb0a12a4c6b7b9936646345a299f1fa4700 Jul 28 17:43:08.964571 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:08.964532 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" event={"ID":"98584ef1-bcb8-45ed-89ed-4397e2be4676","Type":"ContainerStarted","Data":"2c5757f9ad778c330953b1178a59cdeca8ba674facc1adf1ad91cdf2af8f8071"} Jul 28 17:43:08.965928 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:08.965901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" event={"ID":"24639594-3084-4a56-9e0c-6d18b0fa0bc3","Type":"ContainerStarted","Data":"70a6de75159d237782d15a0222e34eb0a12a4c6b7b9936646345a299f1fa4700"} Jul 28 17:43:08.982624 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:08.982580 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-77lsk" podStartSLOduration=2.738506977 podStartE2EDuration="23.982560718s" totalImagesPullingTime="21.244053741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:08.981161141 +0000 UTC m=+513.953668691" watchObservedRunningTime="2026-07-28 17:43:08.982560718 +0000 UTC m=+513.955068267" Jul 28 17:43:11.067574 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:43:11.067522 2576 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-h46g8_kuadrant-system_675c979c-f640-4428-a73e-efe7f017cc95_0(04b98e3d2160e567bf9faeb9eb339cba6c31cc23f7e47cb60f9876146a177ce6): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-h46g8 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 28 17:43:11.067905 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:43:11.067599 2576 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-h46g8_kuadrant-system_675c979c-f640-4428-a73e-efe7f017cc95_0(04b98e3d2160e567bf9faeb9eb339cba6c31cc23f7e47cb60f9876146a177ce6): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-h46g8 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:43:11.981663 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:11.981631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:43:11.985136 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:43:11.985108 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" is forbidden: User \"system:node:ip-10-0-139-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-157.ec2.internal' and this object" podUID="675c979c-f640-4428-a73e-efe7f017cc95" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:43:11.990133 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:11.990111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:43:11.993208 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:43:11.993156 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" is forbidden: User \"system:node:ip-10-0-139-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-157.ec2.internal' and this object" podUID="675c979c-f640-4428-a73e-efe7f017cc95" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:43:12.030729 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:12.030690 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z68d\" (UniqueName: \"kubernetes.io/projected/675c979c-f640-4428-a73e-efe7f017cc95-kube-api-access-6z68d\") pod \"675c979c-f640-4428-a73e-efe7f017cc95\" (UID: \"675c979c-f640-4428-a73e-efe7f017cc95\") " Jul 28 17:43:12.030944 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:12.030787 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/675c979c-f640-4428-a73e-efe7f017cc95-extensions-socket-volume\") pod \"675c979c-f640-4428-a73e-efe7f017cc95\" (UID: \"675c979c-f640-4428-a73e-efe7f017cc95\") " Jul 28 17:43:12.031090 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:12.031064 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675c979c-f640-4428-a73e-efe7f017cc95-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "675c979c-f640-4428-a73e-efe7f017cc95" (UID: "675c979c-f640-4428-a73e-efe7f017cc95"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:43:12.033094 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:12.033058 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675c979c-f640-4428-a73e-efe7f017cc95-kube-api-access-6z68d" (OuterVolumeSpecName: "kube-api-access-6z68d") pod "675c979c-f640-4428-a73e-efe7f017cc95" (UID: "675c979c-f640-4428-a73e-efe7f017cc95"). InnerVolumeSpecName "kube-api-access-6z68d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:43:12.131968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:12.131931 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/675c979c-f640-4428-a73e-efe7f017cc95-extensions-socket-volume\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:43:12.131968 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:12.131966 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z68d\" (UniqueName: \"kubernetes.io/projected/675c979c-f640-4428-a73e-efe7f017cc95-kube-api-access-6z68d\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:43:12.986943 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:12.986914 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:43:12.989274 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:43:12.989235 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" is forbidden: User \"system:node:ip-10-0-139-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-157.ec2.internal' and this object" podUID="675c979c-f640-4428-a73e-efe7f017cc95" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:43:12.995336 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:43:12.995312 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-h46g8\" is forbidden: User \"system:node:ip-10-0-139-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-157.ec2.internal' and this object" podUID="675c979c-f640-4428-a73e-efe7f017cc95" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h46g8" Jul 28 17:43:13.598593 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:13.598555 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="675c979c-f640-4428-a73e-efe7f017cc95" path="/var/lib/kubelet/pods/675c979c-f640-4428-a73e-efe7f017cc95/volumes" Jul 28 17:43:13.993505 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:13.993474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" event={"ID":"24639594-3084-4a56-9e0c-6d18b0fa0bc3","Type":"ContainerStarted","Data":"2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839"} Jul 28 17:43:13.993680 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:13.993564 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" podUID="24639594-3084-4a56-9e0c-6d18b0fa0bc3" containerName="manager" containerID="cri-o://2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839" gracePeriod=10 Jul 28 17:43:13.993680 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:13.993641 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:43:14.013038 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.012977 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" podStartSLOduration=13.836486712 podStartE2EDuration="19.012964533s" totalImagesPullingTime="5.176477821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:14.011779691 +0000 UTC m=+518.984287240" watchObservedRunningTime="2026-07-28 17:43:14.012964533 +0000 UTC m=+518.985472083" Jul 28 17:43:14.248458 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.248434 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:43:14.355548 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.355508 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24639594-3084-4a56-9e0c-6d18b0fa0bc3-extensions-socket-volume\") pod \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\" (UID: \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\") " Jul 28 17:43:14.355696 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.355605 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf9g6\" (UniqueName: \"kubernetes.io/projected/24639594-3084-4a56-9e0c-6d18b0fa0bc3-kube-api-access-xf9g6\") pod \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\" (UID: \"24639594-3084-4a56-9e0c-6d18b0fa0bc3\") " Jul 28 17:43:14.355794 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.355748 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24639594-3084-4a56-9e0c-6d18b0fa0bc3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "24639594-3084-4a56-9e0c-6d18b0fa0bc3" (UID: "24639594-3084-4a56-9e0c-6d18b0fa0bc3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:43:14.357625 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.357607 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24639594-3084-4a56-9e0c-6d18b0fa0bc3-kube-api-access-xf9g6" (OuterVolumeSpecName: "kube-api-access-xf9g6") pod "24639594-3084-4a56-9e0c-6d18b0fa0bc3" (UID: "24639594-3084-4a56-9e0c-6d18b0fa0bc3"). InnerVolumeSpecName "kube-api-access-xf9g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:43:14.456728 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.456683 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9g6\" (UniqueName: \"kubernetes.io/projected/24639594-3084-4a56-9e0c-6d18b0fa0bc3-kube-api-access-xf9g6\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:43:14.456728 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.456723 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24639594-3084-4a56-9e0c-6d18b0fa0bc3-extensions-socket-volume\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:43:14.999056 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.999020 2576 generic.go:356] "Generic (PLEG): container finished" podID="24639594-3084-4a56-9e0c-6d18b0fa0bc3" containerID="2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839" exitCode=2 Jul 28 17:43:14.999491 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.999065 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" Jul 28 17:43:14.999491 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.999060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" event={"ID":"24639594-3084-4a56-9e0c-6d18b0fa0bc3","Type":"ContainerDied","Data":"2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839"} Jul 28 17:43:14.999491 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.999125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k" event={"ID":"24639594-3084-4a56-9e0c-6d18b0fa0bc3","Type":"ContainerDied","Data":"70a6de75159d237782d15a0222e34eb0a12a4c6b7b9936646345a299f1fa4700"} Jul 28 17:43:14.999491 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:14.999147 2576 scope.go:117] "RemoveContainer" containerID="2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839" Jul 28 17:43:15.007595 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:15.007577 2576 scope.go:117] "RemoveContainer" containerID="2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839" Jul 28 17:43:15.007857 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:43:15.007838 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839\": container with ID starting with 2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839 not found: ID does not exist" containerID="2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839" Jul 28 17:43:15.007918 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:15.007870 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839"} err="failed to get container status \"2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839\": rpc error: code = NotFound desc = could not find container \"2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839\": container with ID starting with 2a0ebc316dd5a63598520861f0142f3785133407b84c8fa2935ce4645747a839 not found: ID does not exist" Jul 28 17:43:15.021369 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:15.021348 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k"] Jul 28 17:43:15.026075 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:15.026053 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h6l8k"] Jul 28 17:43:15.596146 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:15.596114 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24639594-3084-4a56-9e0c-6d18b0fa0bc3" path="/var/lib/kubelet/pods/24639594-3084-4a56-9e0c-6d18b0fa0bc3/volumes" Jul 28 17:43:41.712025 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.711991 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:43:41.712514 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.712376 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24639594-3084-4a56-9e0c-6d18b0fa0bc3" containerName="manager" Jul 28 17:43:41.712514 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.712388 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="24639594-3084-4a56-9e0c-6d18b0fa0bc3" containerName="manager" Jul 28 17:43:41.712514 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.712448 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="24639594-3084-4a56-9e0c-6d18b0fa0bc3" containerName="manager" Jul 28 17:43:41.714384 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.714365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:43:41.716709 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.716682 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 17:43:41.725727 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.725706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:43:41.754592 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.754557 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:43:41.809370 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.809328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47479\" (UniqueName: \"kubernetes.io/projected/1c32686d-90b5-4073-906c-629d9ff3b435-kube-api-access-47479\") pod \"limitador-limitador-54d6d988d5-cq5k9\" (UID: \"1c32686d-90b5-4073-906c-629d9ff3b435\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:43:41.809540 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.809485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c32686d-90b5-4073-906c-629d9ff3b435-config-file\") pod \"limitador-limitador-54d6d988d5-cq5k9\" (UID: \"1c32686d-90b5-4073-906c-629d9ff3b435\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:43:41.909948 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.909912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47479\" (UniqueName: \"kubernetes.io/projected/1c32686d-90b5-4073-906c-629d9ff3b435-kube-api-access-47479\") pod \"limitador-limitador-54d6d988d5-cq5k9\" (UID: \"1c32686d-90b5-4073-906c-629d9ff3b435\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:43:41.910143 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.910009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c32686d-90b5-4073-906c-629d9ff3b435-config-file\") pod \"limitador-limitador-54d6d988d5-cq5k9\" (UID: \"1c32686d-90b5-4073-906c-629d9ff3b435\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:43:41.910596 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.910577 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c32686d-90b5-4073-906c-629d9ff3b435-config-file\") pod \"limitador-limitador-54d6d988d5-cq5k9\" (UID: \"1c32686d-90b5-4073-906c-629d9ff3b435\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:43:41.918886 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:41.918860 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47479\" (UniqueName: \"kubernetes.io/projected/1c32686d-90b5-4073-906c-629d9ff3b435-kube-api-access-47479\") pod \"limitador-limitador-54d6d988d5-cq5k9\" (UID: \"1c32686d-90b5-4073-906c-629d9ff3b435\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:43:42.025124 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.025008 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:43:42.160332 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.160310 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:43:42.162771 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:43:42.162744 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c32686d_90b5_4073_906c_629d9ff3b435.slice/crio-0390396c3f666ebcd213bce3262cbab0a9843399f9725ce34c849b7bed546754 WatchSource:0}: Error finding container 0390396c3f666ebcd213bce3262cbab0a9843399f9725ce34c849b7bed546754: Status 404 returned error can't find the container with id 0390396c3f666ebcd213bce3262cbab0a9843399f9725ce34c849b7bed546754 Jul 28 17:43:42.435049 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.435017 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-c6xm7"] Jul 28 17:43:42.437810 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.437793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-c6xm7" Jul 28 17:43:42.440060 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.440041 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bp9pv\"" Jul 28 17:43:42.444480 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.444456 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c6xm7"] Jul 28 17:43:42.615407 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.615371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wz6l\" (UniqueName: \"kubernetes.io/projected/415ace92-f6e5-46e6-ab7c-84c3b29925af-kube-api-access-8wz6l\") pod \"authorino-75d556bd77-c6xm7\" (UID: \"415ace92-f6e5-46e6-ab7c-84c3b29925af\") " pod="kuadrant-system/authorino-75d556bd77-c6xm7" Jul 28 17:43:42.716620 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.716547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wz6l\" (UniqueName: \"kubernetes.io/projected/415ace92-f6e5-46e6-ab7c-84c3b29925af-kube-api-access-8wz6l\") pod \"authorino-75d556bd77-c6xm7\" (UID: \"415ace92-f6e5-46e6-ab7c-84c3b29925af\") " pod="kuadrant-system/authorino-75d556bd77-c6xm7" Jul 28 17:43:42.731597 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.731560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wz6l\" (UniqueName: \"kubernetes.io/projected/415ace92-f6e5-46e6-ab7c-84c3b29925af-kube-api-access-8wz6l\") pod \"authorino-75d556bd77-c6xm7\" (UID: \"415ace92-f6e5-46e6-ab7c-84c3b29925af\") " pod="kuadrant-system/authorino-75d556bd77-c6xm7" Jul 28 17:43:42.748395 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.748364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-c6xm7" Jul 28 17:43:42.936071 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:42.936032 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c6xm7"] Jul 28 17:43:42.942184 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:43:42.941784 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod415ace92_f6e5_46e6_ab7c_84c3b29925af.slice/crio-b27bf43ff3632bd426c67f7a71511c978cc875fa1faf9317e110e76c0775fab7 WatchSource:0}: Error finding container b27bf43ff3632bd426c67f7a71511c978cc875fa1faf9317e110e76c0775fab7: Status 404 returned error can't find the container with id b27bf43ff3632bd426c67f7a71511c978cc875fa1faf9317e110e76c0775fab7 Jul 28 17:43:43.129925 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:43.129829 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-c6xm7" event={"ID":"415ace92-f6e5-46e6-ab7c-84c3b29925af","Type":"ContainerStarted","Data":"b27bf43ff3632bd426c67f7a71511c978cc875fa1faf9317e110e76c0775fab7"} Jul 28 17:43:43.131292 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:43.131260 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" event={"ID":"1c32686d-90b5-4073-906c-629d9ff3b435","Type":"ContainerStarted","Data":"0390396c3f666ebcd213bce3262cbab0a9843399f9725ce34c849b7bed546754"} Jul 28 17:43:47.155162 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:47.155120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-c6xm7" event={"ID":"415ace92-f6e5-46e6-ab7c-84c3b29925af","Type":"ContainerStarted","Data":"3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9"} Jul 28 17:43:47.156811 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:47.156777 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" event={"ID":"1c32686d-90b5-4073-906c-629d9ff3b435","Type":"ContainerStarted","Data":"e359ab2228733289ef363a0a019c612b68273a18cd6d2067b96e3929c947a599"} Jul 28 17:43:47.156951 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:47.156873 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:43:47.175569 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:47.175517 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-c6xm7" podStartSLOduration=1.634003766 podStartE2EDuration="5.175501702s" totalImagesPullingTime="3.541497936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:47.173622528 +0000 UTC m=+552.146130079" watchObservedRunningTime="2026-07-28 17:43:47.175501702 +0000 UTC m=+552.148009249" Jul 28 17:43:47.194657 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:47.194604 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" podStartSLOduration=1.885071937 podStartE2EDuration="6.194583299s" totalImagesPullingTime="4.309511362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:47.193774848 +0000 UTC m=+552.166282399" watchObservedRunningTime="2026-07-28 17:43:47.194583299 +0000 UTC m=+552.167090848" Jul 28 17:43:58.162077 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:43:58.162048 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-cq5k9" Jul 28 17:44:19.715673 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:19.715633 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c6xm7"] Jul 28 17:44:19.716136 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:19.715832 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-c6xm7" podUID="415ace92-f6e5-46e6-ab7c-84c3b29925af" containerName="authorino" containerID="cri-o://3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9" gracePeriod=30 Jul 28 17:44:19.964660 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:19.964636 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-c6xm7" Jul 28 17:44:20.076404 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.076315 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wz6l\" (UniqueName: \"kubernetes.io/projected/415ace92-f6e5-46e6-ab7c-84c3b29925af-kube-api-access-8wz6l\") pod \"415ace92-f6e5-46e6-ab7c-84c3b29925af\" (UID: \"415ace92-f6e5-46e6-ab7c-84c3b29925af\") " Jul 28 17:44:20.078435 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.078406 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415ace92-f6e5-46e6-ab7c-84c3b29925af-kube-api-access-8wz6l" (OuterVolumeSpecName: "kube-api-access-8wz6l") pod "415ace92-f6e5-46e6-ab7c-84c3b29925af" (UID: "415ace92-f6e5-46e6-ab7c-84c3b29925af"). InnerVolumeSpecName "kube-api-access-8wz6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:44:20.177003 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.176952 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wz6l\" (UniqueName: \"kubernetes.io/projected/415ace92-f6e5-46e6-ab7c-84c3b29925af-kube-api-access-8wz6l\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 17:44:20.303755 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.303724 2576 generic.go:356] "Generic (PLEG): container finished" podID="415ace92-f6e5-46e6-ab7c-84c3b29925af" containerID="3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9" exitCode=0 Jul 28 17:44:20.303935 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.303763 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-c6xm7" Jul 28 17:44:20.303935 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.303808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-c6xm7" event={"ID":"415ace92-f6e5-46e6-ab7c-84c3b29925af","Type":"ContainerDied","Data":"3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9"} Jul 28 17:44:20.303935 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.303845 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-c6xm7" event={"ID":"415ace92-f6e5-46e6-ab7c-84c3b29925af","Type":"ContainerDied","Data":"b27bf43ff3632bd426c67f7a71511c978cc875fa1faf9317e110e76c0775fab7"} Jul 28 17:44:20.303935 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.303863 2576 scope.go:117] "RemoveContainer" containerID="3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9" Jul 28 17:44:20.313124 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.313102 2576 scope.go:117] "RemoveContainer" containerID="3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9" Jul 28 17:44:20.313473 ip-10-0-139-157 kubenswrapper[2576]: E0728 17:44:20.313453 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9\": container with ID starting with 3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9 not found: ID does not exist" containerID="3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9" Jul 28 17:44:20.313556 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.313485 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9"} err="failed to get container status \"3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9\": rpc error: code = NotFound desc = could not find container \"3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9\": container with ID starting with 3a548156a7b02c3646de3123c1840e0e0f061038391ee605fcb47befea02a9b9 not found: ID does not exist" Jul 28 17:44:20.325944 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.325919 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c6xm7"] Jul 28 17:44:20.330352 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:20.330300 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c6xm7"] Jul 28 17:44:21.596017 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:44:21.595981 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415ace92-f6e5-46e6-ab7c-84c3b29925af" path="/var/lib/kubelet/pods/415ace92-f6e5-46e6-ab7c-84c3b29925af/volumes" Jul 28 17:45:02.577658 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:02.577623 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:45:24.383160 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.383123 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x"] Jul 28 17:45:24.383740 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.383720 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415ace92-f6e5-46e6-ab7c-84c3b29925af" containerName="authorino" Jul 28 17:45:24.383823 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.383743 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="415ace92-f6e5-46e6-ab7c-84c3b29925af" containerName="authorino" Jul 28 17:45:24.383883 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.383874 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="415ace92-f6e5-46e6-ab7c-84c3b29925af" containerName="authorino" Jul 28 17:45:24.386346 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.386322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.389443 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.389421 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 17:45:24.389648 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.389422 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dc42l\"" Jul 28 17:45:24.389648 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.389525 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 28 17:45:24.389648 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.389422 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 17:45:24.399772 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.399742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x"] Jul 28 17:45:24.454809 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.454769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.454809 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.454816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.455035 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.454844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwjqx\" (UniqueName: \"kubernetes.io/projected/63549de2-af35-4c4d-84e3-bebab67c0de5-kube-api-access-fwjqx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.455035 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.454866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.455035 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.454908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63549de2-af35-4c4d-84e3-bebab67c0de5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.455035 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.454969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.556141 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.556108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.556329 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.556195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63549de2-af35-4c4d-84e3-bebab67c0de5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.556329 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.556236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.556329 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.556272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.556329 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.556315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.556561 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.556348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwjqx\" (UniqueName: \"kubernetes.io/projected/63549de2-af35-4c4d-84e3-bebab67c0de5-kube-api-access-fwjqx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.556619 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.556575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.556697 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.556619 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.556697 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.556647 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.558535 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.558511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63549de2-af35-4c4d-84e3-bebab67c0de5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.558769 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.558752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63549de2-af35-4c4d-84e3-bebab67c0de5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.565501 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.565478 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwjqx\" (UniqueName: \"kubernetes.io/projected/63549de2-af35-4c4d-84e3-bebab67c0de5-kube-api-access-fwjqx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x\" (UID: \"63549de2-af35-4c4d-84e3-bebab67c0de5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.698195 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.698132 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:24.844503 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.844476 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x"] Jul 28 17:45:24.847469 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:45:24.847438 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63549de2_af35_4c4d_84e3_bebab67c0de5.slice/crio-323868cf62ec0a2a9faca1171bb891702aeb6219c8f331f50bd424471485d56b WatchSource:0}: Error finding container 323868cf62ec0a2a9faca1171bb891702aeb6219c8f331f50bd424471485d56b: Status 404 returned error can't find the container with id 323868cf62ec0a2a9faca1171bb891702aeb6219c8f331f50bd424471485d56b Jul 28 17:45:24.848860 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:24.848842 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:45:25.001483 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:25.001398 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:45:25.608807 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:25.608773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" event={"ID":"63549de2-af35-4c4d-84e3-bebab67c0de5","Type":"ContainerStarted","Data":"323868cf62ec0a2a9faca1171bb891702aeb6219c8f331f50bd424471485d56b"} Jul 28 17:45:26.616214 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:26.616158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" event={"ID":"63549de2-af35-4c4d-84e3-bebab67c0de5","Type":"ContainerStarted","Data":"9d58605e700b88b3270eaacfb73a06f6f6b4906824a40722bd9c44b23b3ad708"} Jul 28 17:45:26.637464 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:26.637404 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" podStartSLOduration=1.4502239430000001 podStartE2EDuration="2.637387471s" totalImagesPullingTime="1.187163528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:26.636272937 +0000 UTC m=+651.608780492" watchObservedRunningTime="2026-07-28 17:45:26.637387471 +0000 UTC m=+651.609895022" Jul 28 17:45:34.698978 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:34.698943 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:34.699537 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:34.699004 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:34.711130 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:34.711111 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:35.669647 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:35.669606 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x" Jul 28 17:45:37.014669 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:37.014633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:45:46.009401 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:46.009365 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:45:53.696425 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:45:53.696393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:46:05.010609 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.010573 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj"] Jul 28 17:46:05.013336 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.013318 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.016829 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.016372 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 28 17:46:05.029695 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.029665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj"] Jul 28 17:46:05.147441 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.147402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81631af-8507-4336-9874-1941ac3b9674-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.147441 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.147445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.147650 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.147489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.147650 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.147511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.147650 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.147529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.147749 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.147648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljrh\" (UniqueName: \"kubernetes.io/projected/a81631af-8507-4336-9874-1941ac3b9674-kube-api-access-7ljrh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.249117 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.249066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81631af-8507-4336-9874-1941ac3b9674-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.249117 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.249125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.249396 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.249167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.249396 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.249217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.249396 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.249243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.249396 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.249347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljrh\" (UniqueName: \"kubernetes.io/projected/a81631af-8507-4336-9874-1941ac3b9674-kube-api-access-7ljrh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.249670 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.249640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.249774 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.249691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.249829 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.249784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.251429 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.251408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81631af-8507-4336-9874-1941ac3b9674-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.251722 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.251705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81631af-8507-4336-9874-1941ac3b9674-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.258595 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.258574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljrh\" (UniqueName: \"kubernetes.io/projected/a81631af-8507-4336-9874-1941ac3b9674-kube-api-access-7ljrh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjbhj\" (UID: \"a81631af-8507-4336-9874-1941ac3b9674\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.329429 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.329320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:05.466031 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.466005 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj"] Jul 28 17:46:05.468363 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:46:05.468336 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81631af_8507_4336_9874_1941ac3b9674.slice/crio-e277e0e91698bb9a7b34afd10afe6ce561b3b3decaed07953caccfdfb2712801 WatchSource:0}: Error finding container e277e0e91698bb9a7b34afd10afe6ce561b3b3decaed07953caccfdfb2712801: Status 404 returned error can't find the container with id e277e0e91698bb9a7b34afd10afe6ce561b3b3decaed07953caccfdfb2712801 Jul 28 17:46:05.802710 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.802675 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" event={"ID":"a81631af-8507-4336-9874-1941ac3b9674","Type":"ContainerStarted","Data":"f4d50f9b784366558756025d049ea3bc3e4a3e1ad773141806171eda7aea8d1f"} Jul 28 17:46:05.802912 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.802730 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" event={"ID":"a81631af-8507-4336-9874-1941ac3b9674","Type":"ContainerStarted","Data":"e277e0e91698bb9a7b34afd10afe6ce561b3b3decaed07953caccfdfb2712801"} Jul 28 17:46:05.829239 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:05.829152 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" podStartSLOduration=1.642673219 podStartE2EDuration="1.829132953s" totalImagesPullingTime="186.459734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:05.827603828 +0000 UTC m=+690.800111376" watchObservedRunningTime="2026-07-28 17:46:05.829132953 +0000 UTC m=+690.801640504" Jul 28 17:46:07.603831 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:07.603793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:46:08.203837 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.203806 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6"] Jul 28 17:46:08.206401 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.206380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.211048 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.211023 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 28 17:46:08.223103 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.223043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6"] Jul 28 17:46:08.379775 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.379717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819ec53b-e499-4e79-bc96-5b79b7118628-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.379775 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.379780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.380046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.379902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvntz\" (UniqueName: \"kubernetes.io/projected/819ec53b-e499-4e79-bc96-5b79b7118628-kube-api-access-hvntz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.380046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.379947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.380046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.380009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.380046 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.380027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.480786 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.480692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.480934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.480786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvntz\" (UniqueName: \"kubernetes.io/projected/819ec53b-e499-4e79-bc96-5b79b7118628-kube-api-access-hvntz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.480934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.480815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.480934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.480875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.480934 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.480898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.481165 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.481124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819ec53b-e499-4e79-bc96-5b79b7118628-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.481327 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.481300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.481440 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.481340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.481440 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.481401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.483013 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.482988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819ec53b-e499-4e79-bc96-5b79b7118628-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.483455 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.483434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819ec53b-e499-4e79-bc96-5b79b7118628-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.490312 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.490291 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvntz\" (UniqueName: \"kubernetes.io/projected/819ec53b-e499-4e79-bc96-5b79b7118628-kube-api-access-hvntz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksds6\" (UID: \"819ec53b-e499-4e79-bc96-5b79b7118628\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.516389 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.516349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:08.672308 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.672284 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6"] Jul 28 17:46:08.674397 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:46:08.674365 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod819ec53b_e499_4e79_bc96_5b79b7118628.slice/crio-bf03dfd609d3f9e91f4856bb348d3b562b66fd20a84c8cf00d0caf49cc151e79 WatchSource:0}: Error finding container bf03dfd609d3f9e91f4856bb348d3b562b66fd20a84c8cf00d0caf49cc151e79: Status 404 returned error can't find the container with id bf03dfd609d3f9e91f4856bb348d3b562b66fd20a84c8cf00d0caf49cc151e79 Jul 28 17:46:08.818754 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:08.818669 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" event={"ID":"819ec53b-e499-4e79-bc96-5b79b7118628","Type":"ContainerStarted","Data":"bf03dfd609d3f9e91f4856bb348d3b562b66fd20a84c8cf00d0caf49cc151e79"} Jul 28 17:46:09.824492 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:09.824458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" event={"ID":"819ec53b-e499-4e79-bc96-5b79b7118628","Type":"ContainerStarted","Data":"e8d51d4a796a38dbd681b94edecf136f525dc09101b034ebe0ec2ef8a9d94618"} Jul 28 17:46:09.843397 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:09.843322 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" podStartSLOduration=1.670373223 podStartE2EDuration="1.843307215s" totalImagesPullingTime="172.933992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:09.842886713 +0000 UTC m=+694.815394288" watchObservedRunningTime="2026-07-28 17:46:09.843307215 +0000 UTC m=+694.815814765" Jul 28 17:46:15.330093 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:15.330050 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:15.330546 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:15.330128 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:15.342301 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:15.342275 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:15.866038 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:15.866012 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjbhj" Jul 28 17:46:18.517363 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:18.517328 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:18.517830 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:18.517380 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:18.529955 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:18.529918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:18.880077 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:18.879988 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksds6" Jul 28 17:46:19.393313 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:46:19.393277 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:47:02.269913 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.269874 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4d84c9cf-4lxfd"] Jul 28 17:47:02.272374 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.272356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" Jul 28 17:47:02.275796 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.275769 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bp9pv\"" Jul 28 17:47:02.275796 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.275773 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 28 17:47:02.290947 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.290906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4d84c9cf-4lxfd"] Jul 28 17:47:02.391337 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.391295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f49897b0-6e10-4a02-981a-3000d0763280-tls-cert\") pod \"authorino-6d4d84c9cf-4lxfd\" (UID: \"f49897b0-6e10-4a02-981a-3000d0763280\") " pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" Jul 28 17:47:02.391507 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.391352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5zq8\" (UniqueName: \"kubernetes.io/projected/f49897b0-6e10-4a02-981a-3000d0763280-kube-api-access-j5zq8\") pod \"authorino-6d4d84c9cf-4lxfd\" (UID: \"f49897b0-6e10-4a02-981a-3000d0763280\") " pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" Jul 28 17:47:02.492841 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.492797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5zq8\" (UniqueName: \"kubernetes.io/projected/f49897b0-6e10-4a02-981a-3000d0763280-kube-api-access-j5zq8\") pod \"authorino-6d4d84c9cf-4lxfd\" (UID: \"f49897b0-6e10-4a02-981a-3000d0763280\") " pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" Jul 28 17:47:02.493044 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.492934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f49897b0-6e10-4a02-981a-3000d0763280-tls-cert\") pod \"authorino-6d4d84c9cf-4lxfd\" (UID: \"f49897b0-6e10-4a02-981a-3000d0763280\") " pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" Jul 28 17:47:02.495392 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.495371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f49897b0-6e10-4a02-981a-3000d0763280-tls-cert\") pod \"authorino-6d4d84c9cf-4lxfd\" (UID: \"f49897b0-6e10-4a02-981a-3000d0763280\") " pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" Jul 28 17:47:02.504204 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.504158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5zq8\" (UniqueName: \"kubernetes.io/projected/f49897b0-6e10-4a02-981a-3000d0763280-kube-api-access-j5zq8\") pod \"authorino-6d4d84c9cf-4lxfd\" (UID: \"f49897b0-6e10-4a02-981a-3000d0763280\") " pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" Jul 28 17:47:02.582352 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.582235 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" Jul 28 17:47:02.726064 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:02.726032 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4d84c9cf-4lxfd"] Jul 28 17:47:02.728526 ip-10-0-139-157 kubenswrapper[2576]: W0728 17:47:02.728499 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf49897b0_6e10_4a02_981a_3000d0763280.slice/crio-3be47a896ed255d757b9649b265a4b5f59f169b414a2892579ec16415aaca713 WatchSource:0}: Error finding container 3be47a896ed255d757b9649b265a4b5f59f169b414a2892579ec16415aaca713: Status 404 returned error can't find the container with id 3be47a896ed255d757b9649b265a4b5f59f169b414a2892579ec16415aaca713 Jul 28 17:47:03.075414 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:03.075363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" event={"ID":"f49897b0-6e10-4a02-981a-3000d0763280","Type":"ContainerStarted","Data":"3be47a896ed255d757b9649b265a4b5f59f169b414a2892579ec16415aaca713"} Jul 28 17:47:04.082368 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:04.082327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" event={"ID":"f49897b0-6e10-4a02-981a-3000d0763280","Type":"ContainerStarted","Data":"6ef4c6ff7fa365dc86a8052e0e96b5dd893c0770d03fc5c2d8f154aa6f4f5abc"} Jul 28 17:47:04.111367 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:04.111318 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d4d84c9cf-4lxfd" podStartSLOduration=1.615378298 podStartE2EDuration="2.111297087s" totalImagesPullingTime="495.918789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:04.109106625 +0000 UTC m=+749.081614176" watchObservedRunningTime="2026-07-28 17:47:04.111297087 +0000 UTC m=+749.083804668" Jul 28 17:47:34.404493 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:34.404458 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:47:43.696318 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:43.696283 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:47:53.909646 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:47:53.909611 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:49:39.598633 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:49:39.598598 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:49:48.597482 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:49:48.597442 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:50:01.201693 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:50:01.201661 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:50:19.801106 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:50:19.801066 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:50:30.100189 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:50:30.100152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:50:39.899258 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:50:39.899224 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:50:47.497195 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:50:47.497148 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:50:58.002699 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:50:58.002599 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:51:07.900946 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:51:07.900910 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:51:20.395065 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:51:20.395026 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:51:30.493609 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:51:30.493571 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:51:51.900946 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:51:51.900905 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:52:02.399372 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:52:02.399337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:52:26.193603 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:52:26.193568 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:52:36.797227 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:52:36.797129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:52:50.710554 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:52:50.710511 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:53:04.818999 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:53:04.818963 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:53:16.614507 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:53:16.614468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:53:25.714996 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:53:25.714962 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:53:35.301578 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:53:35.301544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:53:44.316259 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:53:44.316222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:53:56.414544 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:53:56.414507 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:54:04.511522 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:54:04.511485 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:55:45.520263 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:55:45.520223 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:55:56.498298 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:55:56.498259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:56:13.116820 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:56:13.116783 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:56:25.907506 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:56:25.907470 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:56:35.205489 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:56:35.205450 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:56:46.901448 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:56:46.901415 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:56:56.216279 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:56:56.216245 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:57:07.406853 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:57:07.406772 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:57:18.004313 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:57:18.004281 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:57:27.511756 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:57:27.511721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:57:38.002878 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:57:38.002842 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:58:02.501291 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:58:02.501251 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:58:11.114014 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:58:11.113979 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:58:25.699687 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:58:25.699653 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:58:52.108528 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:58:52.108493 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 17:59:53.406973 ip-10-0-139-157 kubenswrapper[2576]: I0728 17:59:53.406936 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:00:06.114997 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:00:06.114917 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:00:15.796193 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:00:15.796149 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:00:27.805119 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:00:27.805083 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:00:37.702816 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:00:37.702773 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:00:46.810235 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:00:46.810198 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:00:57.042103 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:00:57.042065 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:01:06.001940 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:01:06.001902 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:01:15.020570 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:01:15.020539 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:01:29.232205 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:01:29.232104 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:01:38.902998 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:01:38.902962 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:01:48.413841 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:01:48.413792 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:02:00.905907 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:02:00.905869 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:02:12.102541 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:02:12.102507 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:02:20.299903 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:02:20.299867 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:02:38.914330 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:02:38.914294 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:02:51.904801 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:02:51.904768 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:03:02.908050 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:03:02.907965 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:03:11.507304 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:03:11.507268 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:03:20.602023 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:03:20.601989 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:03:42.514136 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:03:42.514097 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:03:59.195677 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:03:59.195643 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:04:27.902904 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:04:27.902820 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:04:38.010904 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:04:38.010864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:05:02.231788 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:05:02.231749 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:05:12.013991 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:05:12.013951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:05:36.929483 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:05:36.929448 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:05:47.119393 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:05:47.119354 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:06:11.506143 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:06:11.506102 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:06:21.827925 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:06:21.827889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:07:39.106686 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:07:39.106596 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:07:55.497476 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:07:55.497433 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:08:05.313493 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:08:05.313453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:08:16.497528 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:08:16.497468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:08:51.897576 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:08:51.897536 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:09:29.597335 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:09:29.597302 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:09:54.705453 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:09:54.705416 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:10:14.497018 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:10:14.496981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:10:36.499374 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:10:36.499296 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:10:46.702052 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:10:46.702010 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:12:02.508792 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:12:02.508708 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:12:13.905543 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:12:13.905505 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:12:23.910723 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:12:23.910686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:13:17.419605 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:13:17.419562 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:13:48.213008 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:13:48.212971 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:14:20.018110 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:20.018076 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:14:31.810487 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:31.810447 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:14:56.511825 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.511787 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k"] Jul 28 18:14:56.515070 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.515043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.517745 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.517710 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-95c506\"/\"openshift-service-ca.crt\"" Jul 28 18:14:56.517745 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.517721 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-95c506\"/\"kube-root-ca.crt\"" Jul 28 18:14:56.518438 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.518414 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-95c506\"/\"default-dockercfg-qbhg8\"" Jul 28 18:14:56.518559 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.518540 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-95c506\"/\"test-model-95c506-kserve-self-signed-certs\"" Jul 28 18:14:56.527301 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.527274 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k"] Jul 28 18:14:56.616529 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.616484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwmb\" (UniqueName: \"kubernetes.io/projected/752c8fa5-ca37-43a2-849f-36082c367cfd-kube-api-access-6kwmb\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.616529 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.616533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-home\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.616768 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.616580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-model-cache\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.616768 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.616650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-dshm\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.616768 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.616695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-tmp-dir\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.616902 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.616766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/752c8fa5-ca37-43a2-849f-36082c367cfd-tls-certs\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.717983 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.717889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwmb\" (UniqueName: \"kubernetes.io/projected/752c8fa5-ca37-43a2-849f-36082c367cfd-kube-api-access-6kwmb\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.717983 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.717935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-home\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.717983 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.717952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-model-cache\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.718317 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.718008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-dshm\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.718317 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.718051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-tmp-dir\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.718317 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.718074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/752c8fa5-ca37-43a2-849f-36082c367cfd-tls-certs\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.718563 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.718540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-home\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.718637 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.718568 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-model-cache\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.718637 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.718597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-tmp-dir\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.720380 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.720358 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-dshm\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.720637 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.720620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/752c8fa5-ca37-43a2-849f-36082c367cfd-tls-certs\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.727760 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.727728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwmb\" (UniqueName: \"kubernetes.io/projected/752c8fa5-ca37-43a2-849f-36082c367cfd-kube-api-access-6kwmb\") pod \"test-model-95c506-kserve-597789dd57-d9g7k\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.825130 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.825031 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:14:56.972935 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:56.972840 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k"] Jul 28 18:14:56.975306 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:14:56.975272 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod752c8fa5_ca37_43a2_849f_36082c367cfd.slice/crio-8519e423465ef88dbb75ac02c90b48452babfaea53bfd504ac69a063b5fe2dc8 WatchSource:0}: Error finding container 8519e423465ef88dbb75ac02c90b48452babfaea53bfd504ac69a063b5fe2dc8: Status 404 returned error can't find the container with id 8519e423465ef88dbb75ac02c90b48452babfaea53bfd504ac69a063b5fe2dc8 Jul 28 18:14:57.345616 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:57.345576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" event={"ID":"752c8fa5-ca37-43a2-849f-36082c367cfd","Type":"ContainerStarted","Data":"2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8"} Jul 28 18:14:57.345799 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:57.345629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" event={"ID":"752c8fa5-ca37-43a2-849f-36082c367cfd","Type":"ContainerStarted","Data":"8519e423465ef88dbb75ac02c90b48452babfaea53bfd504ac69a063b5fe2dc8"} Jul 28 18:14:57.370212 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:14:57.370138 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" podStartSLOduration=1.370117251 podStartE2EDuration="1.370117251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:14:57.367257018 +0000 UTC m=+2422.339764565" watchObservedRunningTime="2026-07-28 18:14:57.370117251 +0000 UTC m=+2422.342624802" Jul 28 18:15:02.719345 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:15:02.719306 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:15:06.825267 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:15:06.825228 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:15:06.825267 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:15:06.825286 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:15:06.837888 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:15:06.837850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:15:07.407206 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:15:07.407158 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:15:56.895609 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:15:56.895571 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k"] Jul 28 18:16:00.806732 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:00.806698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:16:11.915509 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:11.915442 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" podUID="752c8fa5-ca37-43a2-849f-36082c367cfd" containerName="main" containerID="cri-o://2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8" gracePeriod=45 Jul 28 18:16:12.166607 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.166534 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:16:12.354165 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354126 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kwmb\" (UniqueName: \"kubernetes.io/projected/752c8fa5-ca37-43a2-849f-36082c367cfd-kube-api-access-6kwmb\") pod \"752c8fa5-ca37-43a2-849f-36082c367cfd\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " Jul 28 18:16:12.354379 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354243 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-dshm\") pod \"752c8fa5-ca37-43a2-849f-36082c367cfd\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " Jul 28 18:16:12.354379 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354266 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-tmp-dir\") pod \"752c8fa5-ca37-43a2-849f-36082c367cfd\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " Jul 28 18:16:12.354379 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354286 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/752c8fa5-ca37-43a2-849f-36082c367cfd-tls-certs\") pod \"752c8fa5-ca37-43a2-849f-36082c367cfd\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " Jul 28 18:16:12.354379 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354313 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-home\") pod \"752c8fa5-ca37-43a2-849f-36082c367cfd\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " Jul 28 18:16:12.354379 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354331 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-model-cache\") pod \"752c8fa5-ca37-43a2-849f-36082c367cfd\" (UID: \"752c8fa5-ca37-43a2-849f-36082c367cfd\") " Jul 28 18:16:12.354652 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354562 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "752c8fa5-ca37-43a2-849f-36082c367cfd" (UID: "752c8fa5-ca37-43a2-849f-36082c367cfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:16:12.354751 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354727 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-tmp-dir\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:16:12.354751 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354730 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-model-cache" (OuterVolumeSpecName: "model-cache") pod "752c8fa5-ca37-43a2-849f-36082c367cfd" (UID: "752c8fa5-ca37-43a2-849f-36082c367cfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:16:12.354859 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.354742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-home" (OuterVolumeSpecName: "home") pod "752c8fa5-ca37-43a2-849f-36082c367cfd" (UID: "752c8fa5-ca37-43a2-849f-36082c367cfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:16:12.356618 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.356592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-dshm" (OuterVolumeSpecName: "dshm") pod "752c8fa5-ca37-43a2-849f-36082c367cfd" (UID: "752c8fa5-ca37-43a2-849f-36082c367cfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:16:12.356618 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.356593 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752c8fa5-ca37-43a2-849f-36082c367cfd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "752c8fa5-ca37-43a2-849f-36082c367cfd" (UID: "752c8fa5-ca37-43a2-849f-36082c367cfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:16:12.356770 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.356668 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752c8fa5-ca37-43a2-849f-36082c367cfd-kube-api-access-6kwmb" (OuterVolumeSpecName: "kube-api-access-6kwmb") pod "752c8fa5-ca37-43a2-849f-36082c367cfd" (UID: "752c8fa5-ca37-43a2-849f-36082c367cfd"). InnerVolumeSpecName "kube-api-access-6kwmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:16:12.455778 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.455683 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwmb\" (UniqueName: \"kubernetes.io/projected/752c8fa5-ca37-43a2-849f-36082c367cfd-kube-api-access-6kwmb\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:16:12.455778 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.455715 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-dshm\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:16:12.455778 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.455725 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/752c8fa5-ca37-43a2-849f-36082c367cfd-tls-certs\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:16:12.455778 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.455736 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-home\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:16:12.455778 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.455746 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/752c8fa5-ca37-43a2-849f-36082c367cfd-model-cache\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:16:12.734594 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.734501 2576 generic.go:356] "Generic (PLEG): container finished" podID="752c8fa5-ca37-43a2-849f-36082c367cfd" containerID="2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8" exitCode=0 Jul 28 18:16:12.734594 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.734544 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" Jul 28 18:16:12.734794 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.734590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" event={"ID":"752c8fa5-ca37-43a2-849f-36082c367cfd","Type":"ContainerDied","Data":"2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8"} Jul 28 18:16:12.734794 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.734625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k" event={"ID":"752c8fa5-ca37-43a2-849f-36082c367cfd","Type":"ContainerDied","Data":"8519e423465ef88dbb75ac02c90b48452babfaea53bfd504ac69a063b5fe2dc8"} Jul 28 18:16:12.734794 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.734641 2576 scope.go:117] "RemoveContainer" containerID="2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8" Jul 28 18:16:12.744681 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.744653 2576 scope.go:117] "RemoveContainer" containerID="2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8" Jul 28 18:16:12.744977 ip-10-0-139-157 kubenswrapper[2576]: E0728 18:16:12.744956 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8\": container with ID starting with 2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8 not found: ID does not exist" containerID="2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8" Jul 28 18:16:12.745035 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.744986 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8"} err="failed to get container status \"2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8\": rpc error: code = NotFound desc = could not find container \"2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8\": container with ID starting with 2c674961d166ecf654fd409f93e342a6368d874c5ac80d01f542738d091012e8 not found: ID does not exist" Jul 28 18:16:12.758947 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.758909 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k"] Jul 28 18:16:12.762585 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:12.762556 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-95c506/test-model-95c506-kserve-597789dd57-d9g7k"] Jul 28 18:16:13.296842 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:13.296805 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:16:13.597525 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:13.597436 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="752c8fa5-ca37-43a2-849f-36082c367cfd" path="/var/lib/kubelet/pods/752c8fa5-ca37-43a2-849f-36082c367cfd/volumes" Jul 28 18:16:24.799077 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:24.799046 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:16:41.197616 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:16:41.197573 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:17:09.895846 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:17:09.895807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:17:57.823088 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:17:57.823001 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:18:15.109977 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:18:15.109938 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:18:30.207456 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:18:30.207416 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:18:44.099509 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:18:44.099463 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:19:12.801461 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:12.801429 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:19:38.306888 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.306805 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b"] Jul 28 18:19:38.307304 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.307285 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752c8fa5-ca37-43a2-849f-36082c367cfd" containerName="main" Jul 28 18:19:38.307304 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.307300 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="752c8fa5-ca37-43a2-849f-36082c367cfd" containerName="main" Jul 28 18:19:38.307395 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.307371 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="752c8fa5-ca37-43a2-849f-36082c367cfd" containerName="main" Jul 28 18:19:38.310550 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.310533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.312823 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.312802 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c73c1e\"/\"openshift-service-ca.crt\"" Jul 28 18:19:38.312961 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.312909 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c73c1e\"/\"kube-root-ca.crt\"" Jul 28 18:19:38.313682 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.313664 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c73c1e\"/\"default-dockercfg-w24mz\"" Jul 28 18:19:38.313682 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.313678 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c73c1e\"/\"auth-test-model-c73c1e-kserve-self-signed-certs\"" Jul 28 18:19:38.321465 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.321442 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b"] Jul 28 18:19:38.446762 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.446725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tls-certs\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.446762 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.446765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-dshm\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.447001 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.446800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7qx\" (UniqueName: \"kubernetes.io/projected/094f218d-8786-4d5a-92f7-ba0e20e02d4f-kube-api-access-rb7qx\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.447001 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.446865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-home\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.447001 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.446935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tmp-dir\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.447103 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.447021 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-model-cache\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.547672 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.547636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-home\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.547885 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.547688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tmp-dir\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.547885 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.547732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-model-cache\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.547885 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.547784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tls-certs\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.547885 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.547808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-dshm\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.547885 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.547849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7qx\" (UniqueName: \"kubernetes.io/projected/094f218d-8786-4d5a-92f7-ba0e20e02d4f-kube-api-access-rb7qx\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.548162 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.548109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-home\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.548162 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.548146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tmp-dir\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.548318 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.548215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-model-cache\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.550048 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.550029 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-dshm\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.550389 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.550373 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tls-certs\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.557348 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.557277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7qx\" (UniqueName: \"kubernetes.io/projected/094f218d-8786-4d5a-92f7-ba0e20e02d4f-kube-api-access-rb7qx\") pod \"auth-test-model-c73c1e-kserve-cd44548f9-46m5b\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.620058 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.620015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:38.772843 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:38.772769 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b"] Jul 28 18:19:38.775214 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:19:38.775160 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod094f218d_8786_4d5a_92f7_ba0e20e02d4f.slice/crio-00089b7f0dd44ac2adaa0bb95962b9a922cc9966fe1060eeb3d47e9e79a75823 WatchSource:0}: Error finding container 00089b7f0dd44ac2adaa0bb95962b9a922cc9966fe1060eeb3d47e9e79a75823: Status 404 returned error can't find the container with id 00089b7f0dd44ac2adaa0bb95962b9a922cc9966fe1060eeb3d47e9e79a75823 Jul 28 18:19:39.763658 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:39.763625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" event={"ID":"094f218d-8786-4d5a-92f7-ba0e20e02d4f","Type":"ContainerStarted","Data":"51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a"} Jul 28 18:19:39.763658 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:39.763664 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" event={"ID":"094f218d-8786-4d5a-92f7-ba0e20e02d4f","Type":"ContainerStarted","Data":"00089b7f0dd44ac2adaa0bb95962b9a922cc9966fe1060eeb3d47e9e79a75823"} Jul 28 18:19:39.785721 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:39.785658 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" podStartSLOduration=1.785641638 podStartE2EDuration="1.785641638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:19:39.78366447 +0000 UTC m=+2704.756172024" watchObservedRunningTime="2026-07-28 18:19:39.785641638 +0000 UTC m=+2704.758149209" Jul 28 18:19:42.000358 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:42.000305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:19:48.620511 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:48.620463 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:48.621143 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:48.620535 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:48.633765 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:48.633734 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:19:48.822151 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:19:48.822115 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:20:09.898015 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:20:09.897975 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:20:35.994770 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:20:35.994737 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:20:54.103631 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:20:54.103600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:21:17.530163 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:21:17.530126 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:21:33.904315 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:21:33.904282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:21:47.208140 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:21:47.208107 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:22:13.325394 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:22:13.325355 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:22:30.432732 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:22:30.432647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:22:53.228759 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:22:53.228722 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:23:10.806913 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:23:10.806873 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:23:38.923461 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:23:38.923430 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:23:57.211095 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:23:57.211005 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:24:19.299955 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:19.299921 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:24:23.435490 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:23.435457 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b"] Jul 28 18:24:33.631842 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.631803 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l"] Jul 28 18:24:33.635605 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.635582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.637875 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.637857 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9b5b66\"/\"kube-root-ca.crt\"" Jul 28 18:24:33.638587 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.638565 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b5b66\"/\"sub-test-model-9b5b66-kserve-self-signed-certs\"" Jul 28 18:24:33.638682 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.638597 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9b5b66\"/\"openshift-service-ca.crt\"" Jul 28 18:24:33.638682 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.638618 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b5b66\"/\"default-dockercfg-p45rt\"" Jul 28 18:24:33.650155 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.650133 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l"] Jul 28 18:24:33.677575 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.677542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tls-certs\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.677713 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.677600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-model-cache\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.677713 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.677642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-home\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.677822 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.677750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwkb\" (UniqueName: \"kubernetes.io/projected/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-kube-api-access-4bwkb\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.677822 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.677812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tmp-dir\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.677930 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.677839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-dshm\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.779059 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.779024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-home\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.779277 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.779085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwkb\" (UniqueName: \"kubernetes.io/projected/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-kube-api-access-4bwkb\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.779277 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.779111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tmp-dir\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.779277 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.779128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-dshm\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.779277 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.779208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tls-certs\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.779277 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.779239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-model-cache\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.779564 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.779545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-home\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.779612 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.779581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tmp-dir\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.779696 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.779678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-model-cache\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.781422 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.781401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-dshm\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.781654 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.781635 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tls-certs\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.788267 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.788245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwkb\" (UniqueName: \"kubernetes.io/projected/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-kube-api-access-4bwkb\") pod \"sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:33.944860 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:33.944828 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:34.085469 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:34.085445 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l"] Jul 28 18:24:34.087086 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:24:34.087055 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod121eff69_a9dd_4f14_a1bf_cc017b9ae3c0.slice/crio-14dfc181b42df6f9147691dd07066ef9ce8114ec81b5b5ebe9ebfc010f423087 WatchSource:0}: Error finding container 14dfc181b42df6f9147691dd07066ef9ce8114ec81b5b5ebe9ebfc010f423087: Status 404 returned error can't find the container with id 14dfc181b42df6f9147691dd07066ef9ce8114ec81b5b5ebe9ebfc010f423087 Jul 28 18:24:34.213449 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:34.213373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" event={"ID":"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0","Type":"ContainerStarted","Data":"0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b"} Jul 28 18:24:34.213449 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:34.213411 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" event={"ID":"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0","Type":"ContainerStarted","Data":"14dfc181b42df6f9147691dd07066ef9ce8114ec81b5b5ebe9ebfc010f423087"} Jul 28 18:24:34.234704 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:34.234653 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" podStartSLOduration=1.234635554 podStartE2EDuration="1.234635554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:24:34.233819544 +0000 UTC m=+2999.206327096" watchObservedRunningTime="2026-07-28 18:24:34.234635554 +0000 UTC m=+2999.207143105" Jul 28 18:24:36.799283 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:36.798708 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:24:38.454875 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.454833 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" podUID="094f218d-8786-4d5a-92f7-ba0e20e02d4f" containerName="main" containerID="cri-o://51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a" gracePeriod=45 Jul 28 18:24:38.717015 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.716949 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:24:38.824545 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.824512 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-dshm\") pod \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " Jul 28 18:24:38.824545 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.824549 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tmp-dir\") pod \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " Jul 28 18:24:38.824813 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.824566 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-model-cache\") pod \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " Jul 28 18:24:38.824813 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.824585 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-home\") pod \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " Jul 28 18:24:38.824813 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.824633 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb7qx\" (UniqueName: \"kubernetes.io/projected/094f218d-8786-4d5a-92f7-ba0e20e02d4f-kube-api-access-rb7qx\") pod \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " Jul 28 18:24:38.824813 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.824717 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tls-certs\") pod \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\" (UID: \"094f218d-8786-4d5a-92f7-ba0e20e02d4f\") " Jul 28 18:24:38.825032 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.824861 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "094f218d-8786-4d5a-92f7-ba0e20e02d4f" (UID: "094f218d-8786-4d5a-92f7-ba0e20e02d4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:24:38.825032 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.824897 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "094f218d-8786-4d5a-92f7-ba0e20e02d4f" (UID: "094f218d-8786-4d5a-92f7-ba0e20e02d4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:24:38.825032 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.824911 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-home" (OuterVolumeSpecName: "home") pod "094f218d-8786-4d5a-92f7-ba0e20e02d4f" (UID: "094f218d-8786-4d5a-92f7-ba0e20e02d4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:24:38.825157 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.825133 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tmp-dir\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:24:38.825157 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.825154 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-model-cache\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:24:38.825260 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.825163 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-home\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:24:38.826992 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.826967 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-dshm" (OuterVolumeSpecName: "dshm") pod "094f218d-8786-4d5a-92f7-ba0e20e02d4f" (UID: "094f218d-8786-4d5a-92f7-ba0e20e02d4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:24:38.826992 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.826979 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "094f218d-8786-4d5a-92f7-ba0e20e02d4f" (UID: "094f218d-8786-4d5a-92f7-ba0e20e02d4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:24:38.827156 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.826970 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094f218d-8786-4d5a-92f7-ba0e20e02d4f-kube-api-access-rb7qx" (OuterVolumeSpecName: "kube-api-access-rb7qx") pod "094f218d-8786-4d5a-92f7-ba0e20e02d4f" (UID: "094f218d-8786-4d5a-92f7-ba0e20e02d4f"). InnerVolumeSpecName "kube-api-access-rb7qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:24:38.926347 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.926312 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094f218d-8786-4d5a-92f7-ba0e20e02d4f-dshm\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:24:38.926347 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.926343 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7qx\" (UniqueName: \"kubernetes.io/projected/094f218d-8786-4d5a-92f7-ba0e20e02d4f-kube-api-access-rb7qx\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:24:38.926561 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:38.926358 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094f218d-8786-4d5a-92f7-ba0e20e02d4f-tls-certs\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:24:39.240018 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.239987 2576 generic.go:356] "Generic (PLEG): container finished" podID="094f218d-8786-4d5a-92f7-ba0e20e02d4f" containerID="51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a" exitCode=0 Jul 28 18:24:39.240243 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.240029 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" Jul 28 18:24:39.240243 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.240061 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" event={"ID":"094f218d-8786-4d5a-92f7-ba0e20e02d4f","Type":"ContainerDied","Data":"51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a"} Jul 28 18:24:39.240243 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.240101 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b" event={"ID":"094f218d-8786-4d5a-92f7-ba0e20e02d4f","Type":"ContainerDied","Data":"00089b7f0dd44ac2adaa0bb95962b9a922cc9966fe1060eeb3d47e9e79a75823"} Jul 28 18:24:39.240243 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.240120 2576 scope.go:117] "RemoveContainer" containerID="51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a" Jul 28 18:24:39.249395 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.249378 2576 scope.go:117] "RemoveContainer" containerID="51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a" Jul 28 18:24:39.249651 ip-10-0-139-157 kubenswrapper[2576]: E0728 18:24:39.249634 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a\": container with ID starting with 51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a not found: ID does not exist" containerID="51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a" Jul 28 18:24:39.249701 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.249661 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a"} err="failed to get container status \"51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a\": rpc error: code = NotFound desc = could not find container \"51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a\": container with ID starting with 51d491bda3759e5d1d7cc4d4ba1cf6b3b7bfa070b7a2b5ea1940472d0148cb7a not found: ID does not exist" Jul 28 18:24:39.264899 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.264867 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b"] Jul 28 18:24:39.266600 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.266581 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c73c1e/auth-test-model-c73c1e-kserve-cd44548f9-46m5b"] Jul 28 18:24:39.596282 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:39.596204 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094f218d-8786-4d5a-92f7-ba0e20e02d4f" path="/var/lib/kubelet/pods/094f218d-8786-4d5a-92f7-ba0e20e02d4f/volumes" Jul 28 18:24:43.945581 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:43.945542 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:43.946090 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:43.945594 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:43.958045 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:43.958009 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:24:44.277993 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:24:44.277904 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:25:02.806599 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:25:02.806566 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:25:32.494789 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:25:32.494753 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:25:53.597898 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:25:53.597869 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:25:58.905000 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:25:58.904966 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l"] Jul 28 18:26:04.095564 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.095525 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89"] Jul 28 18:26:04.096185 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.096150 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094f218d-8786-4d5a-92f7-ba0e20e02d4f" containerName="main" Jul 28 18:26:04.096244 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.096199 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="094f218d-8786-4d5a-92f7-ba0e20e02d4f" containerName="main" Jul 28 18:26:04.096346 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.096333 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="094f218d-8786-4d5a-92f7-ba0e20e02d4f" containerName="main" Jul 28 18:26:04.100027 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.100006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.102372 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.102350 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b5b66\"/\"rate-test-model-9b5b66-kserve-self-signed-certs\"" Jul 28 18:26:04.107047 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.107020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6nmm\" (UniqueName: \"kubernetes.io/projected/76a59860-2a06-4f2c-aa7a-29e596909b39-kube-api-access-t6nmm\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.107557 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.107450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a59860-2a06-4f2c-aa7a-29e596909b39-tls-certs\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.107632 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.107576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-dshm\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.107632 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.107601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-tmp-dir\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.107743 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.107652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-model-cache\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.107743 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.107725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-home\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.108452 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.108428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89"] Jul 28 18:26:04.208672 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.208636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-dshm\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.208672 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.208674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-tmp-dir\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.208968 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.208710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-model-cache\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.208968 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.208751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-home\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.208968 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.208799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6nmm\" (UniqueName: \"kubernetes.io/projected/76a59860-2a06-4f2c-aa7a-29e596909b39-kube-api-access-t6nmm\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.208968 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.208830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a59860-2a06-4f2c-aa7a-29e596909b39-tls-certs\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.209240 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.209220 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-tmp-dir\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.209317 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.209262 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-model-cache\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.209317 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.209304 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-home\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.211107 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.211069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-dshm\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.211262 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.211240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a59860-2a06-4f2c-aa7a-29e596909b39-tls-certs\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.218896 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.218857 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6nmm\" (UniqueName: \"kubernetes.io/projected/76a59860-2a06-4f2c-aa7a-29e596909b39-kube-api-access-t6nmm\") pod \"rate-test-model-9b5b66-kserve-854df98f7f-pfq89\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.410954 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.410914 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:04.547851 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.547827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89"] Jul 28 18:26:04.550632 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:26:04.550597 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76a59860_2a06_4f2c_aa7a_29e596909b39.slice/crio-228dce02a37eb88bb49a00530e37345565b5f2e85deea680a8242e2353c14e59 WatchSource:0}: Error finding container 228dce02a37eb88bb49a00530e37345565b5f2e85deea680a8242e2353c14e59: Status 404 returned error can't find the container with id 228dce02a37eb88bb49a00530e37345565b5f2e85deea680a8242e2353c14e59 Jul 28 18:26:04.654216 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.654155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" event={"ID":"76a59860-2a06-4f2c-aa7a-29e596909b39","Type":"ContainerStarted","Data":"935debba3d292f31a561ed3c7e08d430bd4a137bbfb5dce1049c721ec3a79a7f"} Jul 28 18:26:04.654216 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.654221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" event={"ID":"76a59860-2a06-4f2c-aa7a-29e596909b39","Type":"ContainerStarted","Data":"228dce02a37eb88bb49a00530e37345565b5f2e85deea680a8242e2353c14e59"} Jul 28 18:26:04.674104 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:04.673971 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" podStartSLOduration=0.673950366 podStartE2EDuration="673.950366ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:26:04.673432721 +0000 UTC m=+3089.645940273" watchObservedRunningTime="2026-07-28 18:26:04.673950366 +0000 UTC m=+3089.646457936" Jul 28 18:26:12.996821 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:12.996774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:26:13.923887 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:13.923844 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" podUID="121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" containerName="main" containerID="cri-o://0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b" gracePeriod=45 Jul 28 18:26:14.184259 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.184166 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:26:14.305374 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.305325 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-dshm\") pod \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " Jul 28 18:26:14.305586 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.305445 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bwkb\" (UniqueName: \"kubernetes.io/projected/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-kube-api-access-4bwkb\") pod \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " Jul 28 18:26:14.305586 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.305485 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-model-cache\") pod \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " Jul 28 18:26:14.305586 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.305502 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-home\") pod \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " Jul 28 18:26:14.305586 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.305525 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tls-certs\") pod \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " Jul 28 18:26:14.305814 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.305605 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tmp-dir\") pod \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\" (UID: \"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0\") " Jul 28 18:26:14.305814 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.305798 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-home" (OuterVolumeSpecName: "home") pod "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" (UID: "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:26:14.305932 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.305814 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" (UID: "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:26:14.305932 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.305864 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" (UID: "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:26:14.306046 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.306030 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-model-cache\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:26:14.306104 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.306052 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-home\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:26:14.306104 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.306064 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tmp-dir\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:26:14.307683 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.307659 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-dshm" (OuterVolumeSpecName: "dshm") pod "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" (UID: "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:26:14.307818 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.307694 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" (UID: "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:26:14.307818 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.307790 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-kube-api-access-4bwkb" (OuterVolumeSpecName: "kube-api-access-4bwkb") pod "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" (UID: "121eff69-a9dd-4f14-a1bf-cc017b9ae3c0"). InnerVolumeSpecName "kube-api-access-4bwkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:26:14.407311 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.407275 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-tls-certs\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:26:14.407311 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.407307 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-dshm\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:26:14.407311 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.407316 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwkb\" (UniqueName: \"kubernetes.io/projected/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0-kube-api-access-4bwkb\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:26:14.411465 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.411441 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:14.411631 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.411481 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:14.424398 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.424365 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:14.705053 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.705020 2576 generic.go:356] "Generic (PLEG): container finished" podID="121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" containerID="0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b" exitCode=0 Jul 28 18:26:14.705262 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.705071 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" Jul 28 18:26:14.705262 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.705108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" event={"ID":"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0","Type":"ContainerDied","Data":"0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b"} Jul 28 18:26:14.705262 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.705163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l" event={"ID":"121eff69-a9dd-4f14-a1bf-cc017b9ae3c0","Type":"ContainerDied","Data":"14dfc181b42df6f9147691dd07066ef9ce8114ec81b5b5ebe9ebfc010f423087"} Jul 28 18:26:14.705262 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.705198 2576 scope.go:117] "RemoveContainer" containerID="0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b" Jul 28 18:26:14.715442 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.715418 2576 scope.go:117] "RemoveContainer" containerID="0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b" Jul 28 18:26:14.715751 ip-10-0-139-157 kubenswrapper[2576]: E0728 18:26:14.715730 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b\": container with ID starting with 0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b not found: ID does not exist" containerID="0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b" Jul 28 18:26:14.715817 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.715764 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b"} err="failed to get container status \"0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b\": rpc error: code = NotFound desc = could not find container \"0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b\": container with ID starting with 0927c019b99da76f540778314d40d34a470e72e18b5eac28f045217fe7f8b49b not found: ID does not exist" Jul 28 18:26:14.718112 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.718091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:26:14.728420 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.728395 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l"] Jul 28 18:26:14.735891 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:14.735860 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/sub-test-model-9b5b66-kserve-7b96f6694c-n8t7l"] Jul 28 18:26:15.596602 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:15.596568 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" path="/var/lib/kubelet/pods/121eff69-a9dd-4f14-a1bf-cc017b9ae3c0/volumes" Jul 28 18:26:30.899449 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:26:30.899412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:27:03.598544 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:27:03.598463 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:27:21.597096 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:27:21.597061 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:27:43.032286 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:27:43.032251 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:27:49.009270 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:27:49.009227 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89"] Jul 28 18:27:56.493418 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:27:56.493382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:28:04.029303 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.029254 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" podUID="76a59860-2a06-4f2c-aa7a-29e596909b39" containerName="main" containerID="cri-o://935debba3d292f31a561ed3c7e08d430bd4a137bbfb5dce1049c721ec3a79a7f" gracePeriod=45 Jul 28 18:28:04.255869 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.255838 2576 generic.go:356] "Generic (PLEG): container finished" podID="76a59860-2a06-4f2c-aa7a-29e596909b39" containerID="935debba3d292f31a561ed3c7e08d430bd4a137bbfb5dce1049c721ec3a79a7f" exitCode=0 Jul 28 18:28:04.256027 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.255919 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" event={"ID":"76a59860-2a06-4f2c-aa7a-29e596909b39","Type":"ContainerDied","Data":"935debba3d292f31a561ed3c7e08d430bd4a137bbfb5dce1049c721ec3a79a7f"} Jul 28 18:28:04.291357 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.291282 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:28:04.388659 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.388623 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-model-cache\") pod \"76a59860-2a06-4f2c-aa7a-29e596909b39\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " Jul 28 18:28:04.388864 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.388686 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-tmp-dir\") pod \"76a59860-2a06-4f2c-aa7a-29e596909b39\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " Jul 28 18:28:04.388864 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.388761 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a59860-2a06-4f2c-aa7a-29e596909b39-tls-certs\") pod \"76a59860-2a06-4f2c-aa7a-29e596909b39\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " Jul 28 18:28:04.388864 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.388793 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-home\") pod \"76a59860-2a06-4f2c-aa7a-29e596909b39\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " Jul 28 18:28:04.388864 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.388832 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-dshm\") pod \"76a59860-2a06-4f2c-aa7a-29e596909b39\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " Jul 28 18:28:04.389086 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.388934 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6nmm\" (UniqueName: \"kubernetes.io/projected/76a59860-2a06-4f2c-aa7a-29e596909b39-kube-api-access-t6nmm\") pod \"76a59860-2a06-4f2c-aa7a-29e596909b39\" (UID: \"76a59860-2a06-4f2c-aa7a-29e596909b39\") " Jul 28 18:28:04.389086 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.388975 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-model-cache" (OuterVolumeSpecName: "model-cache") pod "76a59860-2a06-4f2c-aa7a-29e596909b39" (UID: "76a59860-2a06-4f2c-aa7a-29e596909b39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:28:04.389086 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.388982 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76a59860-2a06-4f2c-aa7a-29e596909b39" (UID: "76a59860-2a06-4f2c-aa7a-29e596909b39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:28:04.389247 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.389128 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-home" (OuterVolumeSpecName: "home") pod "76a59860-2a06-4f2c-aa7a-29e596909b39" (UID: "76a59860-2a06-4f2c-aa7a-29e596909b39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:28:04.389327 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.389310 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-model-cache\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:28:04.389382 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.389334 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-tmp-dir\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:28:04.389382 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.389344 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-home\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:28:04.390896 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.390869 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a59860-2a06-4f2c-aa7a-29e596909b39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76a59860-2a06-4f2c-aa7a-29e596909b39" (UID: "76a59860-2a06-4f2c-aa7a-29e596909b39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:28:04.391038 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.390910 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-dshm" (OuterVolumeSpecName: "dshm") pod "76a59860-2a06-4f2c-aa7a-29e596909b39" (UID: "76a59860-2a06-4f2c-aa7a-29e596909b39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:28:04.391038 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.390998 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a59860-2a06-4f2c-aa7a-29e596909b39-kube-api-access-t6nmm" (OuterVolumeSpecName: "kube-api-access-t6nmm") pod "76a59860-2a06-4f2c-aa7a-29e596909b39" (UID: "76a59860-2a06-4f2c-aa7a-29e596909b39"). InnerVolumeSpecName "kube-api-access-t6nmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:28:04.490122 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.490082 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6nmm\" (UniqueName: \"kubernetes.io/projected/76a59860-2a06-4f2c-aa7a-29e596909b39-kube-api-access-t6nmm\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:28:04.490122 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.490118 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a59860-2a06-4f2c-aa7a-29e596909b39-tls-certs\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:28:04.490122 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:04.490132 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a59860-2a06-4f2c-aa7a-29e596909b39-dshm\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:28:05.261759 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.261731 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" Jul 28 18:28:05.261759 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.261745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89" event={"ID":"76a59860-2a06-4f2c-aa7a-29e596909b39","Type":"ContainerDied","Data":"228dce02a37eb88bb49a00530e37345565b5f2e85deea680a8242e2353c14e59"} Jul 28 18:28:05.262328 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.261791 2576 scope.go:117] "RemoveContainer" containerID="935debba3d292f31a561ed3c7e08d430bd4a137bbfb5dce1049c721ec3a79a7f" Jul 28 18:28:05.284026 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.283994 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89"] Jul 28 18:28:05.290420 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.290396 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9b5b66/rate-test-model-9b5b66-kserve-854df98f7f-pfq89"] Jul 28 18:28:05.598906 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.598825 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76a59860-2a06-4f2c-aa7a-29e596909b39" path="/var/lib/kubelet/pods/76a59860-2a06-4f2c-aa7a-29e596909b39/volumes" Jul 28 18:28:05.749852 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.749810 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7734-139928809630720"] Jul 28 18:28:05.750231 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.750217 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a59860-2a06-4f2c-aa7a-29e596909b39" containerName="main" Jul 28 18:28:05.750280 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.750232 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a59860-2a06-4f2c-aa7a-29e596909b39" containerName="main" Jul 28 18:28:05.750280 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.750250 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" containerName="main" Jul 28 18:28:05.750280 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.750257 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" containerName="main" Jul 28 18:28:05.750383 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.750324 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a59860-2a06-4f2c-aa7a-29e596909b39" containerName="main" Jul 28 18:28:05.750383 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.750335 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="121eff69-a9dd-4f14-a1bf-cc017b9ae3c0" containerName="main" Jul 28 18:28:05.754829 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.754805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-139928809630720" Jul 28 18:28:05.757251 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.757223 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-87q2r\"" Jul 28 18:28:05.759342 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.759319 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-139928809630720"] Jul 28 18:28:05.801119 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.801071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9wn\" (UniqueName: \"kubernetes.io/projected/6a65d32a-f63f-4305-8f0c-3a40ff2c424f-kube-api-access-8t9wn\") pod \"test-curl-7734-139928809630720\" (UID: \"6a65d32a-f63f-4305-8f0c-3a40ff2c424f\") " pod="opendatahub/test-curl-7734-139928809630720" Jul 28 18:28:05.902124 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.902087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9wn\" (UniqueName: \"kubernetes.io/projected/6a65d32a-f63f-4305-8f0c-3a40ff2c424f-kube-api-access-8t9wn\") pod \"test-curl-7734-139928809630720\" (UID: \"6a65d32a-f63f-4305-8f0c-3a40ff2c424f\") " pod="opendatahub/test-curl-7734-139928809630720" Jul 28 18:28:05.911590 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:05.911557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9wn\" (UniqueName: \"kubernetes.io/projected/6a65d32a-f63f-4305-8f0c-3a40ff2c424f-kube-api-access-8t9wn\") pod \"test-curl-7734-139928809630720\" (UID: \"6a65d32a-f63f-4305-8f0c-3a40ff2c424f\") " pod="opendatahub/test-curl-7734-139928809630720" Jul 28 18:28:06.065133 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:06.065089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-139928809630720" Jul 28 18:28:06.202630 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:06.202588 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-139928809630720"] Jul 28 18:28:06.205834 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:28:06.205807 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a65d32a_f63f_4305_8f0c_3a40ff2c424f.slice/crio-32605604ee6c35e683c63dcf4cc45d8e586eb721bd686af029b76dece78122e4 WatchSource:0}: Error finding container 32605604ee6c35e683c63dcf4cc45d8e586eb721bd686af029b76dece78122e4: Status 404 returned error can't find the container with id 32605604ee6c35e683c63dcf4cc45d8e586eb721bd686af029b76dece78122e4 Jul 28 18:28:06.207449 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:06.207434 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:28:06.267506 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:06.267474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-139928809630720" event={"ID":"6a65d32a-f63f-4305-8f0c-3a40ff2c424f","Type":"ContainerStarted","Data":"32605604ee6c35e683c63dcf4cc45d8e586eb721bd686af029b76dece78122e4"} Jul 28 18:28:07.274823 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:07.274731 2576 generic.go:356] "Generic (PLEG): container finished" podID="6a65d32a-f63f-4305-8f0c-3a40ff2c424f" containerID="2f40eec40ed58a1e816c0e2ffa3afa93236a8a7bb8b6a7e38feceae9bdeae328" exitCode=0 Jul 28 18:28:07.274823 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:07.274781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-139928809630720" event={"ID":"6a65d32a-f63f-4305-8f0c-3a40ff2c424f","Type":"ContainerDied","Data":"2f40eec40ed58a1e816c0e2ffa3afa93236a8a7bb8b6a7e38feceae9bdeae328"} Jul 28 18:28:08.414770 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:08.414742 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-139928809630720" Jul 28 18:28:08.455453 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:08.455419 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7734-139928809630720_6a65d32a-f63f-4305-8f0c-3a40ff2c424f/test-curl-7734-139928809630720/0.log" Jul 28 18:28:08.483297 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:08.483266 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7734-139928809630720"] Jul 28 18:28:08.485702 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:08.485675 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7734-139928809630720"] Jul 28 18:28:08.528485 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:08.528453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t9wn\" (UniqueName: \"kubernetes.io/projected/6a65d32a-f63f-4305-8f0c-3a40ff2c424f-kube-api-access-8t9wn\") pod \"6a65d32a-f63f-4305-8f0c-3a40ff2c424f\" (UID: \"6a65d32a-f63f-4305-8f0c-3a40ff2c424f\") " Jul 28 18:28:08.530554 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:08.530492 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a65d32a-f63f-4305-8f0c-3a40ff2c424f-kube-api-access-8t9wn" (OuterVolumeSpecName: "kube-api-access-8t9wn") pod "6a65d32a-f63f-4305-8f0c-3a40ff2c424f" (UID: "6a65d32a-f63f-4305-8f0c-3a40ff2c424f"). InnerVolumeSpecName "kube-api-access-8t9wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:28:08.630019 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:08.629981 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t9wn\" (UniqueName: \"kubernetes.io/projected/6a65d32a-f63f-4305-8f0c-3a40ff2c424f-kube-api-access-8t9wn\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:28:09.286351 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:09.286319 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-139928809630720" Jul 28 18:28:09.286539 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:09.286320 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32605604ee6c35e683c63dcf4cc45d8e586eb721bd686af029b76dece78122e4" Jul 28 18:28:09.296421 ip-10-0-139-157 kubenswrapper[2576]: E0728 18:28:09.296385 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7734-139928809630720\" is forbidden: User \"system:node:ip-10-0-139-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-157.ec2.internal' and this object" podUID="6a65d32a-f63f-4305-8f0c-3a40ff2c424f" pod="opendatahub/test-curl-7734-139928809630720" Jul 28 18:28:09.595968 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:09.595888 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a65d32a-f63f-4305-8f0c-3a40ff2c424f" path="/var/lib/kubelet/pods/6a65d32a-f63f-4305-8f0c-3a40ff2c424f/volumes" Jul 28 18:28:21.596928 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:21.596889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:28:52.140204 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:28:52.140152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:29:18.686825 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.686785 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2"] Jul 28 18:29:18.687443 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.687423 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a65d32a-f63f-4305-8f0c-3a40ff2c424f" containerName="test-curl-7734-139928809630720" Jul 28 18:29:18.687526 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.687446 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a65d32a-f63f-4305-8f0c-3a40ff2c424f" containerName="test-curl-7734-139928809630720" Jul 28 18:29:18.687578 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.687549 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a65d32a-f63f-4305-8f0c-3a40ff2c424f" containerName="test-curl-7734-139928809630720" Jul 28 18:29:18.690603 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.690582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.692806 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.692782 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-faea92\"/\"kube-root-ca.crt\"" Jul 28 18:29:18.693412 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.693389 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-faea92\"/\"ipp-route-faea92-kserve-self-signed-certs\"" Jul 28 18:29:18.693516 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.693432 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-faea92\"/\"default-dockercfg-4wqds\"" Jul 28 18:29:18.693516 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.693436 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-faea92\"/\"openshift-service-ca.crt\"" Jul 28 18:29:18.701538 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.701516 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2"] Jul 28 18:29:18.826864 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.826826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tls-certs\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.827041 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.826883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-dshm\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.827041 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.826945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-home\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.827041 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.826962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57k25\" (UniqueName: \"kubernetes.io/projected/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-kube-api-access-57k25\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.827151 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.827059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-model-cache\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.827151 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.827104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tmp-dir\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.928548 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.928515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-model-cache\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.928718 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.928560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tmp-dir\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.928718 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.928610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tls-certs\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.928718 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.928641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-dshm\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.928718 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.928686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-home\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.928718 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.928710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57k25\" (UniqueName: \"kubernetes.io/projected/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-kube-api-access-57k25\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.928989 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.928883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-model-cache\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.928989 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.928960 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tmp-dir\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.929097 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.929027 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-home\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.930869 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.930841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-dshm\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.931091 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.931074 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tls-certs\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:18.936716 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:18.936694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57k25\" (UniqueName: \"kubernetes.io/projected/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-kube-api-access-57k25\") pod \"ipp-route-faea92-kserve-779dc555bc-w7nl2\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:19.000937 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:19.000839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:19.142219 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:19.142194 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2"] Jul 28 18:29:19.144480 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:29:19.144455 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9c1c7c5_0ce2_48ba_b34c_4e0877b17d9a.slice/crio-db5e3ea7ab36f9737e5ef2c16163fb175591c9c2ee1ac36b2afb4d4a33190ead WatchSource:0}: Error finding container db5e3ea7ab36f9737e5ef2c16163fb175591c9c2ee1ac36b2afb4d4a33190ead: Status 404 returned error can't find the container with id db5e3ea7ab36f9737e5ef2c16163fb175591c9c2ee1ac36b2afb4d4a33190ead Jul 28 18:29:19.624696 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:19.624663 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" event={"ID":"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a","Type":"ContainerStarted","Data":"b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc"} Jul 28 18:29:19.624696 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:19.624701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" event={"ID":"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a","Type":"ContainerStarted","Data":"db5e3ea7ab36f9737e5ef2c16163fb175591c9c2ee1ac36b2afb4d4a33190ead"} Jul 28 18:29:19.645634 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:19.645578 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" podStartSLOduration=1.6455614729999999 podStartE2EDuration="1.645561473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:29:19.643818316 +0000 UTC m=+3284.616325865" watchObservedRunningTime="2026-07-28 18:29:19.645561473 +0000 UTC m=+3284.618069024" Jul 28 18:29:22.819258 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:22.819219 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:29:29.001917 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:29.001879 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:29.002342 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:29.002032 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:29.014831 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:29.014800 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:29.690028 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:29.689993 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:29:49.109395 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:29:49.109357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:30:00.157061 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.156932 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v"] Jul 28 18:30:00.161758 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.161723 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck"] Jul 28 18:30:00.162042 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.162016 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v" Jul 28 18:30:00.165742 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.165707 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xcv2\"" Jul 28 18:30:00.166973 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.166947 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482"] Jul 28 18:30:00.167314 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.167295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck" Jul 28 18:30:00.176825 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.176764 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck"] Jul 28 18:30:00.176825 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.176811 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v"] Jul 28 18:30:00.177028 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.176839 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482"] Jul 28 18:30:00.177028 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.176947 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482" Jul 28 18:30:00.180261 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.180238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v" Jul 28 18:30:00.190985 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.190960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck" Jul 28 18:30:00.225548 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.225355 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482" Jul 28 18:30:00.352207 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.352153 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v"] Jul 28 18:30:00.355591 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:30:00.355560 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163ba11f_9b80_4a92_aa6c_5586dc3e51e0.slice/crio-e8471990efb3f7aa3cebda5663371e51e869cf2aa9b46105096ebc018bb8bc64 WatchSource:0}: Error finding container e8471990efb3f7aa3cebda5663371e51e869cf2aa9b46105096ebc018bb8bc64: Status 404 returned error can't find the container with id e8471990efb3f7aa3cebda5663371e51e869cf2aa9b46105096ebc018bb8bc64 Jul 28 18:30:00.409116 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.409037 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482"] Jul 28 18:30:00.412184 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:30:00.412137 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9a9e62_e27f_4dc0_9948_37b9df96244f.slice/crio-f8d29fad97abe19102136a291e9040df33ca5ab109d6a723d7eb45e488935611 WatchSource:0}: Error finding container f8d29fad97abe19102136a291e9040df33ca5ab109d6a723d7eb45e488935611: Status 404 returned error can't find the container with id f8d29fad97abe19102136a291e9040df33ca5ab109d6a723d7eb45e488935611 Jul 28 18:30:00.584548 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.584523 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck"] Jul 28 18:30:00.586635 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:30:00.586607 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c0f9af_7252_472c_8bdd_1d33d275c552.slice/crio-faf27525e703bb63061b76341d9bcbc97f54cb092996d8b9b170db2c7e7308fc WatchSource:0}: Error finding container faf27525e703bb63061b76341d9bcbc97f54cb092996d8b9b170db2c7e7308fc: Status 404 returned error can't find the container with id faf27525e703bb63061b76341d9bcbc97f54cb092996d8b9b170db2c7e7308fc Jul 28 18:30:00.836429 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.836335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482" event={"ID":"1c9a9e62-e27f-4dc0-9948-37b9df96244f","Type":"ContainerStarted","Data":"f8d29fad97abe19102136a291e9040df33ca5ab109d6a723d7eb45e488935611"} Jul 28 18:30:00.837588 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.837556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v" event={"ID":"163ba11f-9b80-4a92-aa6c-5586dc3e51e0","Type":"ContainerStarted","Data":"e8471990efb3f7aa3cebda5663371e51e869cf2aa9b46105096ebc018bb8bc64"} Jul 28 18:30:00.838793 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:00.838764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck" event={"ID":"41c0f9af-7252-472c-8bdd-1d33d275c552","Type":"ContainerStarted","Data":"faf27525e703bb63061b76341d9bcbc97f54cb092996d8b9b170db2c7e7308fc"} Jul 28 18:30:03.858743 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:03.858710 2576 generic.go:356] "Generic (PLEG): container finished" podID="1c9a9e62-e27f-4dc0-9948-37b9df96244f" containerID="94d805eb45e41ac15205200dfa66b00b5fd5a1604c15a378e09ee1a44e8623b0" exitCode=0 Jul 28 18:30:03.859137 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:03.858786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482" event={"ID":"1c9a9e62-e27f-4dc0-9948-37b9df96244f","Type":"ContainerDied","Data":"94d805eb45e41ac15205200dfa66b00b5fd5a1604c15a378e09ee1a44e8623b0"} Jul 28 18:30:03.860343 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:03.860320 2576 generic.go:356] "Generic (PLEG): container finished" podID="41c0f9af-7252-472c-8bdd-1d33d275c552" containerID="b743485d73c972886516393436630a6a94851ec43d59a63beb5131c90090cb96" exitCode=0 Jul 28 18:30:03.860456 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:03.860351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck" event={"ID":"41c0f9af-7252-472c-8bdd-1d33d275c552","Type":"ContainerDied","Data":"b743485d73c972886516393436630a6a94851ec43d59a63beb5131c90090cb96"} Jul 28 18:30:05.034638 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:05.034613 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck" Jul 28 18:30:05.037851 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:05.037822 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482" Jul 28 18:30:05.874020 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:05.873922 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482" Jul 28 18:30:05.874020 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:05.873952 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-4q482" event={"ID":"1c9a9e62-e27f-4dc0-9948-37b9df96244f","Type":"ContainerDied","Data":"f8d29fad97abe19102136a291e9040df33ca5ab109d6a723d7eb45e488935611"} Jul 28 18:30:05.874020 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:05.874011 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8d29fad97abe19102136a291e9040df33ca5ab109d6a723d7eb45e488935611" Jul 28 18:30:05.875336 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:05.875312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck" event={"ID":"41c0f9af-7252-472c-8bdd-1d33d275c552","Type":"ContainerDied","Data":"faf27525e703bb63061b76341d9bcbc97f54cb092996d8b9b170db2c7e7308fc"} Jul 28 18:30:05.875465 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:05.875351 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck" Jul 28 18:30:05.875465 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:05.875357 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faf27525e703bb63061b76341d9bcbc97f54cb092996d8b9b170db2c7e7308fc" Jul 28 18:30:07.887755 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:07.887666 2576 generic.go:356] "Generic (PLEG): container finished" podID="163ba11f-9b80-4a92-aa6c-5586dc3e51e0" containerID="1f4e90c0f80ac0eb1808d672d1de5afb85c886627b5eff2a5c4573d2e2197004" exitCode=0 Jul 28 18:30:07.888142 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:07.887747 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v" event={"ID":"163ba11f-9b80-4a92-aa6c-5586dc3e51e0","Type":"ContainerDied","Data":"1f4e90c0f80ac0eb1808d672d1de5afb85c886627b5eff2a5c4573d2e2197004"} Jul 28 18:30:08.503144 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:08.503109 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:30:09.026928 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:09.026898 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v" Jul 28 18:30:09.899184 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:09.899157 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v" Jul 28 18:30:09.899328 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:09.899200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v" event={"ID":"163ba11f-9b80-4a92-aa6c-5586dc3e51e0","Type":"ContainerDied","Data":"e8471990efb3f7aa3cebda5663371e51e869cf2aa9b46105096ebc018bb8bc64"} Jul 28 18:30:09.899328 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:09.899245 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8471990efb3f7aa3cebda5663371e51e869cf2aa9b46105096ebc018bb8bc64" Jul 28 18:30:18.942957 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.942922 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5"] Jul 28 18:30:18.943522 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.943461 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c0f9af-7252-472c-8bdd-1d33d275c552" containerName="cleanup" Jul 28 18:30:18.943522 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.943479 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c0f9af-7252-472c-8bdd-1d33d275c552" containerName="cleanup" Jul 28 18:30:18.943522 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.943498 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9a9e62-e27f-4dc0-9948-37b9df96244f" containerName="cleanup" Jul 28 18:30:18.943522 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.943507 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9a9e62-e27f-4dc0-9948-37b9df96244f" containerName="cleanup" Jul 28 18:30:18.943522 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.943517 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163ba11f-9b80-4a92-aa6c-5586dc3e51e0" containerName="cleanup" Jul 28 18:30:18.943522 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.943527 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="163ba11f-9b80-4a92-aa6c-5586dc3e51e0" containerName="cleanup" Jul 28 18:30:18.943753 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.943607 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9a9e62-e27f-4dc0-9948-37b9df96244f" containerName="cleanup" Jul 28 18:30:18.943753 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.943620 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c0f9af-7252-472c-8bdd-1d33d275c552" containerName="cleanup" Jul 28 18:30:18.943753 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.943627 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="163ba11f-9b80-4a92-aa6c-5586dc3e51e0" containerName="cleanup" Jul 28 18:30:18.946831 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.946810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" Jul 28 18:30:18.949200 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.949150 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:30:18.949338 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.949286 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xcv2\"" Jul 28 18:30:18.954964 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:18.954941 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5"] Jul 28 18:30:19.018603 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:19.018571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cea3712-db07-4104-9950-15bf1593e4fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5\" (UID: \"7cea3712-db07-4104-9950-15bf1593e4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" Jul 28 18:30:19.119268 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:19.119230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cea3712-db07-4104-9950-15bf1593e4fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5\" (UID: \"7cea3712-db07-4104-9950-15bf1593e4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" Jul 28 18:30:19.119857 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:19.119838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cea3712-db07-4104-9950-15bf1593e4fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5\" (UID: \"7cea3712-db07-4104-9950-15bf1593e4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" Jul 28 18:30:19.258896 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:19.258807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" Jul 28 18:30:19.406165 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:19.406132 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5"] Jul 28 18:30:19.409510 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:30:19.409485 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cea3712_db07_4104_9950_15bf1593e4fe.slice/crio-0627d1bf4c4a11a10301bf1e5ed9a2c17eed406a3bb7c80f0a40beade91c8d6f WatchSource:0}: Error finding container 0627d1bf4c4a11a10301bf1e5ed9a2c17eed406a3bb7c80f0a40beade91c8d6f: Status 404 returned error can't find the container with id 0627d1bf4c4a11a10301bf1e5ed9a2c17eed406a3bb7c80f0a40beade91c8d6f Jul 28 18:30:19.954614 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:19.954515 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" event={"ID":"7cea3712-db07-4104-9950-15bf1593e4fe","Type":"ContainerStarted","Data":"d3675aadabc19e9c4fa49777d59f2b275f72bd584d1dd07fcfd4f1da22ca03a5"} Jul 28 18:30:19.954614 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:19.954574 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" event={"ID":"7cea3712-db07-4104-9950-15bf1593e4fe","Type":"ContainerStarted","Data":"0627d1bf4c4a11a10301bf1e5ed9a2c17eed406a3bb7c80f0a40beade91c8d6f"} Jul 28 18:30:19.971135 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:19.971087 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" podStartSLOduration=1.9710721169999998 podStartE2EDuration="1.971072117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:19.970855604 +0000 UTC m=+3344.943363174" watchObservedRunningTime="2026-07-28 18:30:19.971072117 +0000 UTC m=+3344.943579667" Jul 28 18:30:20.961188 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:20.961150 2576 generic.go:356] "Generic (PLEG): container finished" podID="7cea3712-db07-4104-9950-15bf1593e4fe" containerID="d3675aadabc19e9c4fa49777d59f2b275f72bd584d1dd07fcfd4f1da22ca03a5" exitCode=0 Jul 28 18:30:20.961598 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:20.961234 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" event={"ID":"7cea3712-db07-4104-9950-15bf1593e4fe","Type":"ContainerDied","Data":"d3675aadabc19e9c4fa49777d59f2b275f72bd584d1dd07fcfd4f1da22ca03a5"} Jul 28 18:30:22.094207 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:22.094144 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" Jul 28 18:30:22.148625 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:22.148594 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cea3712-db07-4104-9950-15bf1593e4fe-maas-api-service-ca\") pod \"7cea3712-db07-4104-9950-15bf1593e4fe\" (UID: \"7cea3712-db07-4104-9950-15bf1593e4fe\") " Jul 28 18:30:22.148864 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:22.148842 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cea3712-db07-4104-9950-15bf1593e4fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7cea3712-db07-4104-9950-15bf1593e4fe" (UID: "7cea3712-db07-4104-9950-15bf1593e4fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:30:22.250367 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:22.250274 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cea3712-db07-4104-9950-15bf1593e4fe-maas-api-service-ca\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:30:22.973169 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:22.973134 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" event={"ID":"7cea3712-db07-4104-9950-15bf1593e4fe","Type":"ContainerDied","Data":"0627d1bf4c4a11a10301bf1e5ed9a2c17eed406a3bb7c80f0a40beade91c8d6f"} Jul 28 18:30:22.973395 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:22.973192 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5" Jul 28 18:30:22.973395 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:22.973203 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0627d1bf4c4a11a10301bf1e5ed9a2c17eed406a3bb7c80f0a40beade91c8d6f" Jul 28 18:30:26.087754 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:26.087700 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5"] Jul 28 18:30:26.089622 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:26.089597 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2ad79-8wmg5"] Jul 28 18:30:27.596532 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:27.596499 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cea3712-db07-4104-9950-15bf1593e4fe" path="/var/lib/kubelet/pods/7cea3712-db07-4104-9950-15bf1593e4fe/volumes" Jul 28 18:30:35.292441 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:35.292404 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:30:37.608749 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.608715 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb"] Jul 28 18:30:37.609144 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.609131 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cea3712-db07-4104-9950-15bf1593e4fe" containerName="revoke-keys" Jul 28 18:30:37.609144 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.609144 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cea3712-db07-4104-9950-15bf1593e4fe" containerName="revoke-keys" Jul 28 18:30:37.609258 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.609219 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cea3712-db07-4104-9950-15bf1593e4fe" containerName="revoke-keys" Jul 28 18:30:37.616893 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.616876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" Jul 28 18:30:37.618959 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.618933 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xcv2\"" Jul 28 18:30:37.619472 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.619450 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:30:37.620671 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.620642 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb"] Jul 28 18:30:37.706510 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.706467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df48222-2c6f-4e41-871c-29967dd6edeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb\" (UID: \"8df48222-2c6f-4e41-871c-29967dd6edeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" Jul 28 18:30:37.807353 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.807308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df48222-2c6f-4e41-871c-29967dd6edeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb\" (UID: \"8df48222-2c6f-4e41-871c-29967dd6edeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" Jul 28 18:30:37.807952 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.807931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df48222-2c6f-4e41-871c-29967dd6edeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb\" (UID: \"8df48222-2c6f-4e41-871c-29967dd6edeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" Jul 28 18:30:37.931472 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:37.931435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" Jul 28 18:30:38.081513 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:38.081487 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb"] Jul 28 18:30:38.083878 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:30:38.083851 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df48222_2c6f_4e41_871c_29967dd6edeb.slice/crio-7c382debdecc10594957182dd5b332634e22b57ef4f74d04eeb35edf5ac437a2 WatchSource:0}: Error finding container 7c382debdecc10594957182dd5b332634e22b57ef4f74d04eeb35edf5ac437a2: Status 404 returned error can't find the container with id 7c382debdecc10594957182dd5b332634e22b57ef4f74d04eeb35edf5ac437a2 Jul 28 18:30:39.057817 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:39.057731 2576 generic.go:356] "Generic (PLEG): container finished" podID="8df48222-2c6f-4e41-871c-29967dd6edeb" containerID="94f77879dc291a4d4bebde505190132c5af75d44366a2a37ef9d130d55b92ccd" exitCode=0 Jul 28 18:30:39.057817 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:39.057765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" event={"ID":"8df48222-2c6f-4e41-871c-29967dd6edeb","Type":"ContainerDied","Data":"94f77879dc291a4d4bebde505190132c5af75d44366a2a37ef9d130d55b92ccd"} Jul 28 18:30:39.057817 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:39.057802 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" event={"ID":"8df48222-2c6f-4e41-871c-29967dd6edeb","Type":"ContainerStarted","Data":"7c382debdecc10594957182dd5b332634e22b57ef4f74d04eeb35edf5ac437a2"} Jul 28 18:30:40.193861 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:40.193833 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" Jul 28 18:30:40.331933 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:40.331840 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df48222-2c6f-4e41-871c-29967dd6edeb-maas-api-service-ca\") pod \"8df48222-2c6f-4e41-871c-29967dd6edeb\" (UID: \"8df48222-2c6f-4e41-871c-29967dd6edeb\") " Jul 28 18:30:40.332241 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:40.332218 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df48222-2c6f-4e41-871c-29967dd6edeb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8df48222-2c6f-4e41-871c-29967dd6edeb" (UID: "8df48222-2c6f-4e41-871c-29967dd6edeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:30:40.432697 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:40.432654 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df48222-2c6f-4e41-871c-29967dd6edeb-maas-api-service-ca\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:30:41.069729 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:41.069698 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" Jul 28 18:30:41.069903 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:41.069733 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb" event={"ID":"8df48222-2c6f-4e41-871c-29967dd6edeb","Type":"ContainerDied","Data":"7c382debdecc10594957182dd5b332634e22b57ef4f74d04eeb35edf5ac437a2"} Jul 28 18:30:41.069903 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:41.069791 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c382debdecc10594957182dd5b332634e22b57ef4f74d04eeb35edf5ac437a2" Jul 28 18:30:53.272824 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:53.272774 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb"] Jul 28 18:30:53.274699 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:53.274677 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-faea92-rcmfb"] Jul 28 18:30:53.596695 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:53.596614 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8df48222-2c6f-4e41-871c-29967dd6edeb" path="/var/lib/kubelet/pods/8df48222-2c6f-4e41-871c-29967dd6edeb/volumes" Jul 28 18:30:53.894286 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:53.894256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:30:58.562261 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:30:58.562223 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2"] Jul 28 18:31:04.696766 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:04.696732 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:31:13.582361 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.582298 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" podUID="e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" containerName="main" containerID="cri-o://b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc" gracePeriod=45 Jul 28 18:31:13.836372 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.836309 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:31:13.974470 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.974432 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-home\") pod \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " Jul 28 18:31:13.974470 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.974469 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-dshm\") pod \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " Jul 28 18:31:13.974720 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.974528 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57k25\" (UniqueName: \"kubernetes.io/projected/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-kube-api-access-57k25\") pod \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " Jul 28 18:31:13.974720 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.974567 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tmp-dir\") pod \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " Jul 28 18:31:13.974720 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.974592 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-model-cache\") pod \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " Jul 28 18:31:13.974889 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.974726 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tls-certs\") pod \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\" (UID: \"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a\") " Jul 28 18:31:13.974889 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.974813 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-home" (OuterVolumeSpecName: "home") pod "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" (UID: "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:31:13.974889 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.974870 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" (UID: "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:31:13.975039 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.974884 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" (UID: "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:31:13.975164 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.975146 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-home\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:31:13.975268 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.975169 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tmp-dir\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:31:13.975268 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.975212 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-model-cache\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:31:13.976771 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.976751 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" (UID: "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:31:13.976771 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.976761 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-dshm" (OuterVolumeSpecName: "dshm") pod "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" (UID: "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:31:13.976897 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:13.976851 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-kube-api-access-57k25" (OuterVolumeSpecName: "kube-api-access-57k25") pod "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" (UID: "e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a"). InnerVolumeSpecName "kube-api-access-57k25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:31:14.075731 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.075689 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-tls-certs\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:31:14.075731 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.075721 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-dshm\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:31:14.075731 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.075733 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57k25\" (UniqueName: \"kubernetes.io/projected/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a-kube-api-access-57k25\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:31:14.239092 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.239058 2576 generic.go:356] "Generic (PLEG): container finished" podID="e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" containerID="b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc" exitCode=0 Jul 28 18:31:14.239286 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.239107 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" Jul 28 18:31:14.239286 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.239135 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" event={"ID":"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a","Type":"ContainerDied","Data":"b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc"} Jul 28 18:31:14.239286 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.239165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2" event={"ID":"e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a","Type":"ContainerDied","Data":"db5e3ea7ab36f9737e5ef2c16163fb175591c9c2ee1ac36b2afb4d4a33190ead"} Jul 28 18:31:14.239286 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.239208 2576 scope.go:117] "RemoveContainer" containerID="b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc" Jul 28 18:31:14.248287 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.248267 2576 scope.go:117] "RemoveContainer" containerID="b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc" Jul 28 18:31:14.248541 ip-10-0-139-157 kubenswrapper[2576]: E0728 18:31:14.248519 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc\": container with ID starting with b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc not found: ID does not exist" containerID="b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc" Jul 28 18:31:14.248599 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.248549 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc"} err="failed to get container status \"b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc\": rpc error: code = NotFound desc = could not find container \"b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc\": container with ID starting with b21bec1a23e01f07d84dde3f898c9c294f883ee89348e4c5a02a76b760432efc not found: ID does not exist" Jul 28 18:31:14.261607 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.261580 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2"] Jul 28 18:31:14.266309 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:14.266287 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-faea92/ipp-route-faea92-kserve-779dc555bc-w7nl2"] Jul 28 18:31:15.601877 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:15.601844 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" path="/var/lib/kubelet/pods/e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a/volumes" Jul 28 18:31:44.005761 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.005727 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97"] Jul 28 18:31:44.006397 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.006352 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" containerName="main" Jul 28 18:31:44.006397 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.006373 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" containerName="main" Jul 28 18:31:44.006542 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.006407 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df48222-2c6f-4e41-871c-29967dd6edeb" containerName="revoke-keys" Jul 28 18:31:44.006542 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.006416 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df48222-2c6f-4e41-871c-29967dd6edeb" containerName="revoke-keys" Jul 28 18:31:44.006542 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.006509 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df48222-2c6f-4e41-871c-29967dd6edeb" containerName="revoke-keys" Jul 28 18:31:44.006542 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.006523 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9c1c7c5-0ce2-48ba-b34c-4e0877b17d9a" containerName="main" Jul 28 18:31:44.010791 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.010761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" Jul 28 18:31:44.013688 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.013663 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:31:44.013957 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.013938 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xcv2\"" Jul 28 18:31:44.017849 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.017815 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97"] Jul 28 18:31:44.065450 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.065415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51eaaa6e-69d0-4705-a88c-cc1864e29495-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d53588-khv97\" (UID: \"51eaaa6e-69d0-4705-a88c-cc1864e29495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" Jul 28 18:31:44.166726 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.166689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51eaaa6e-69d0-4705-a88c-cc1864e29495-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d53588-khv97\" (UID: \"51eaaa6e-69d0-4705-a88c-cc1864e29495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" Jul 28 18:31:44.167407 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.167387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51eaaa6e-69d0-4705-a88c-cc1864e29495-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d53588-khv97\" (UID: \"51eaaa6e-69d0-4705-a88c-cc1864e29495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" Jul 28 18:31:44.322958 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.322860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" Jul 28 18:31:44.461276 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:44.461246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97"] Jul 28 18:31:44.463317 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:31:44.463287 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51eaaa6e_69d0_4705_a88c_cc1864e29495.slice/crio-ce200d57c8eb479ed13cae0a5b7f5bdeada3c4d39f6463fa4ab29d3acada47d6 WatchSource:0}: Error finding container ce200d57c8eb479ed13cae0a5b7f5bdeada3c4d39f6463fa4ab29d3acada47d6: Status 404 returned error can't find the container with id ce200d57c8eb479ed13cae0a5b7f5bdeada3c4d39f6463fa4ab29d3acada47d6 Jul 28 18:31:45.400020 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:45.399982 2576 generic.go:356] "Generic (PLEG): container finished" podID="51eaaa6e-69d0-4705-a88c-cc1864e29495" containerID="c6120476287645d57df1a674149156210442d023b751dedb43deb8e277b9cf23" exitCode=0 Jul 28 18:31:45.400443 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:45.400053 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" event={"ID":"51eaaa6e-69d0-4705-a88c-cc1864e29495","Type":"ContainerDied","Data":"c6120476287645d57df1a674149156210442d023b751dedb43deb8e277b9cf23"} Jul 28 18:31:45.400443 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:45.400095 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" event={"ID":"51eaaa6e-69d0-4705-a88c-cc1864e29495","Type":"ContainerStarted","Data":"ce200d57c8eb479ed13cae0a5b7f5bdeada3c4d39f6463fa4ab29d3acada47d6"} Jul 28 18:31:46.537710 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:46.537686 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" Jul 28 18:31:46.689273 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:46.689151 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51eaaa6e-69d0-4705-a88c-cc1864e29495-maas-api-service-ca\") pod \"51eaaa6e-69d0-4705-a88c-cc1864e29495\" (UID: \"51eaaa6e-69d0-4705-a88c-cc1864e29495\") " Jul 28 18:31:46.689578 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:46.689558 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51eaaa6e-69d0-4705-a88c-cc1864e29495-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51eaaa6e-69d0-4705-a88c-cc1864e29495" (UID: "51eaaa6e-69d0-4705-a88c-cc1864e29495"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:31:46.790080 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:46.790040 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51eaaa6e-69d0-4705-a88c-cc1864e29495-maas-api-service-ca\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:31:47.412006 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:47.411972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" event={"ID":"51eaaa6e-69d0-4705-a88c-cc1864e29495","Type":"ContainerDied","Data":"ce200d57c8eb479ed13cae0a5b7f5bdeada3c4d39f6463fa4ab29d3acada47d6"} Jul 28 18:31:47.412006 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:47.412004 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97" Jul 28 18:31:47.412253 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:47.412048 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce200d57c8eb479ed13cae0a5b7f5bdeada3c4d39f6463fa4ab29d3acada47d6" Jul 28 18:31:51.492696 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:51.492662 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck"] Jul 28 18:31:51.495313 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:51.495281 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d53588-29754390-222ck"] Jul 28 18:31:51.596693 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:51.596659 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41c0f9af-7252-472c-8bdd-1d33d275c552" path="/var/lib/kubelet/pods/41c0f9af-7252-472c-8bdd-1d33d275c552/volumes" Jul 28 18:31:51.690107 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:51.690075 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97"] Jul 28 18:31:51.695819 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:51.695779 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d53588-khv97"] Jul 28 18:31:53.596448 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:53.596412 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51eaaa6e-69d0-4705-a88c-cc1864e29495" path="/var/lib/kubelet/pods/51eaaa6e-69d0-4705-a88c-cc1864e29495/volumes" Jul 28 18:31:54.987701 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:54.987665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:31:59.611990 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.611949 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p"] Jul 28 18:31:59.612502 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.612404 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51eaaa6e-69d0-4705-a88c-cc1864e29495" containerName="revoke-keys" Jul 28 18:31:59.612502 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.612416 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="51eaaa6e-69d0-4705-a88c-cc1864e29495" containerName="revoke-keys" Jul 28 18:31:59.612502 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.612487 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="51eaaa6e-69d0-4705-a88c-cc1864e29495" containerName="revoke-keys" Jul 28 18:31:59.615848 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.615831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" Jul 28 18:31:59.618018 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.617995 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xcv2\"" Jul 28 18:31:59.618147 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.618038 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:31:59.622163 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.622135 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p"] Jul 28 18:31:59.717893 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.717856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6956230-d574-44ac-87ff-cd5768a1ccda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p\" (UID: \"a6956230-d574-44ac-87ff-cd5768a1ccda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" Jul 28 18:31:59.819281 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.819246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6956230-d574-44ac-87ff-cd5768a1ccda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p\" (UID: \"a6956230-d574-44ac-87ff-cd5768a1ccda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" Jul 28 18:31:59.819890 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.819872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6956230-d574-44ac-87ff-cd5768a1ccda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p\" (UID: \"a6956230-d574-44ac-87ff-cd5768a1ccda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" Jul 28 18:31:59.929112 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:31:59.929071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" Jul 28 18:32:00.062889 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:00.062854 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p"] Jul 28 18:32:00.065530 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:32:00.065499 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6956230_d574_44ac_87ff_cd5768a1ccda.slice/crio-ef352743d3d4ada0e61c0e7fe9347c8ea00d97d989b8e3ab0d949d74aa7cfccc WatchSource:0}: Error finding container ef352743d3d4ada0e61c0e7fe9347c8ea00d97d989b8e3ab0d949d74aa7cfccc: Status 404 returned error can't find the container with id ef352743d3d4ada0e61c0e7fe9347c8ea00d97d989b8e3ab0d949d74aa7cfccc Jul 28 18:32:00.484888 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:00.484851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" event={"ID":"a6956230-d574-44ac-87ff-cd5768a1ccda","Type":"ContainerStarted","Data":"2fbe064084681610d8fb15a2cfbe308669ab89a346944e8cc7e8fc6f11c7f58d"} Jul 28 18:32:00.485118 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:00.484942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" event={"ID":"a6956230-d574-44ac-87ff-cd5768a1ccda","Type":"ContainerStarted","Data":"ef352743d3d4ada0e61c0e7fe9347c8ea00d97d989b8e3ab0d949d74aa7cfccc"} Jul 28 18:32:00.500672 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:00.500616 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" podStartSLOduration=1.500601606 podStartE2EDuration="1.500601606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:32:00.499884275 +0000 UTC m=+3445.472391826" watchObservedRunningTime="2026-07-28 18:32:00.500601606 +0000 UTC m=+3445.473109158" Jul 28 18:32:01.491234 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:01.491195 2576 generic.go:356] "Generic (PLEG): container finished" podID="a6956230-d574-44ac-87ff-cd5768a1ccda" containerID="2fbe064084681610d8fb15a2cfbe308669ab89a346944e8cc7e8fc6f11c7f58d" exitCode=0 Jul 28 18:32:01.491234 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:01.491237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" event={"ID":"a6956230-d574-44ac-87ff-cd5768a1ccda","Type":"ContainerDied","Data":"2fbe064084681610d8fb15a2cfbe308669ab89a346944e8cc7e8fc6f11c7f58d"} Jul 28 18:32:02.635577 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:02.635550 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" Jul 28 18:32:02.647476 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:02.647448 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6956230-d574-44ac-87ff-cd5768a1ccda-maas-api-service-ca\") pod \"a6956230-d574-44ac-87ff-cd5768a1ccda\" (UID: \"a6956230-d574-44ac-87ff-cd5768a1ccda\") " Jul 28 18:32:02.647783 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:02.647761 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6956230-d574-44ac-87ff-cd5768a1ccda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a6956230-d574-44ac-87ff-cd5768a1ccda" (UID: "a6956230-d574-44ac-87ff-cd5768a1ccda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:32:02.748442 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:02.748359 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6956230-d574-44ac-87ff-cd5768a1ccda-maas-api-service-ca\") on node \"ip-10-0-139-157.ec2.internal\" DevicePath \"\"" Jul 28 18:32:03.503097 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:03.503067 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" Jul 28 18:32:03.503307 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:03.503106 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p" event={"ID":"a6956230-d574-44ac-87ff-cd5768a1ccda","Type":"ContainerDied","Data":"ef352743d3d4ada0e61c0e7fe9347c8ea00d97d989b8e3ab0d949d74aa7cfccc"} Jul 28 18:32:03.503307 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:03.503154 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef352743d3d4ada0e61c0e7fe9347c8ea00d97d989b8e3ab0d949d74aa7cfccc" Jul 28 18:32:07.842050 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:07.842014 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v"] Jul 28 18:32:07.845382 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:07.845354 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9b5b66-29754390-8bl6v"] Jul 28 18:32:08.186618 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:08.186576 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p"] Jul 28 18:32:08.189857 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:08.189830 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9b5b66-wbs4p"] Jul 28 18:32:09.596105 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:09.596069 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163ba11f-9b80-4a92-aa6c-5586dc3e51e0" path="/var/lib/kubelet/pods/163ba11f-9b80-4a92-aa6c-5586dc3e51e0/volumes" Jul 28 18:32:09.596510 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:09.596457 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6956230-d574-44ac-87ff-cd5768a1ccda" path="/var/lib/kubelet/pods/a6956230-d574-44ac-87ff-cd5768a1ccda/volumes" Jul 28 18:32:10.791588 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:10.791545 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:32:24.585466 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:24.585432 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cq5k9"] Jul 28 18:32:29.618397 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:29.618361 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d4d84c9cf-4lxfd_f49897b0-6e10-4a02-981a-3000d0763280/authorino/0.log" Jul 28 18:32:36.155559 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:36.155529 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7c5665b5-sc52l_1d066b5a-7b39-4c48-b5c1-58bb3befbd85/copy-manifests/0.log" Jul 28 18:32:36.162694 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:36.162665 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7c5665b5-sc52l_1d066b5a-7b39-4c48-b5c1-58bb3befbd85/manager/0.log" Jul 28 18:32:36.867062 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:36.867034 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754390-4q482_1c9a9e62-e27f-4dc0-9948-37b9df96244f/cleanup/0.log" Jul 28 18:32:38.400300 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:38.400270 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d4d84c9cf-4lxfd_f49897b0-6e10-4a02-981a-3000d0763280/authorino/0.log" Jul 28 18:32:38.806880 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:38.806804 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-77lsk_98584ef1-bcb8-45ed-89ed-4397e2be4676/kuadrant-console-plugin/0.log" Jul 28 18:32:39.184102 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:39.184067 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-cq5k9_1c32686d-90b5-4073-906c-629d9ff3b435/limitador/0.log" Jul 28 18:32:40.093216 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:40.093165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b8f9cc6c-ssgcc_287d6281-c6a8-4cdf-a0c9-187778f10002/kube-auth-proxy/0.log" Jul 28 18:32:40.997316 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:40.997285 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-kjg9x_63549de2-af35-4c4d-84e3-bebab67c0de5/main/0.log" Jul 28 18:32:41.489819 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:41.489785 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hjbhj_a81631af-8507-4336-9874-1941ac3b9674/main/0.log" Jul 28 18:32:41.610968 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:41.610938 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-ksds6_819ec53b-e499-4e79-bc96-5b79b7118628/main/0.log" Jul 28 18:32:50.742568 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:50.742536 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zp4hx_ff31c586-d5f6-44de-bf64-b6539584c26d/global-pull-secret-syncer/0.log" Jul 28 18:32:50.827832 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:50.827801 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l7q7r_3bcbad70-e788-408a-8f1b-15c7bffafcfe/konnectivity-agent/0.log" Jul 28 18:32:50.923866 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:50.923828 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-157.ec2.internal_49e9400fa5e85fa5702adbac4852ba2c/haproxy/0.log" Jul 28 18:32:56.223142 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:56.223111 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d4d84c9cf-4lxfd_f49897b0-6e10-4a02-981a-3000d0763280/authorino/0.log" Jul 28 18:32:56.332385 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:56.332353 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-77lsk_98584ef1-bcb8-45ed-89ed-4397e2be4676/kuadrant-console-plugin/0.log" Jul 28 18:32:56.613698 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:56.613610 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-cq5k9_1c32686d-90b5-4073-906c-629d9ff3b435/limitador/0.log" Jul 28 18:32:58.485951 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.485870 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d693787-0acb-4c04-9f64-ae5b88cdf424/alertmanager/0.log" Jul 28 18:32:58.515820 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.515785 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d693787-0acb-4c04-9f64-ae5b88cdf424/config-reloader/0.log" Jul 28 18:32:58.543462 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.543436 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d693787-0acb-4c04-9f64-ae5b88cdf424/kube-rbac-proxy-web/0.log" Jul 28 18:32:58.567839 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.567798 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d693787-0acb-4c04-9f64-ae5b88cdf424/kube-rbac-proxy/0.log" Jul 28 18:32:58.591555 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.591529 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d693787-0acb-4c04-9f64-ae5b88cdf424/kube-rbac-proxy-metric/0.log" Jul 28 18:32:58.615600 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.615572 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d693787-0acb-4c04-9f64-ae5b88cdf424/prom-label-proxy/0.log" Jul 28 18:32:58.641347 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.641313 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d693787-0acb-4c04-9f64-ae5b88cdf424/init-config-reloader/0.log" Jul 28 18:32:58.719494 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.719469 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6g7nh_ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2/kube-state-metrics/0.log" Jul 28 18:32:58.758482 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.758409 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6g7nh_ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2/kube-rbac-proxy-main/0.log" Jul 28 18:32:58.797244 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.797169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6g7nh_ab1fe921-33ad-4e0f-bfdb-8af1a6cc9fa2/kube-rbac-proxy-self/0.log" Jul 28 18:32:58.863163 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.863138 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f947b7b5b-qdrhs_42744d32-298e-42d4-9cc7-5b4b38f1dbf2/metrics-server/0.log" Jul 28 18:32:58.893732 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.893699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-2hcng_c37bb557-15c5-4322-98cb-333737867adf/monitoring-plugin/0.log" Jul 28 18:32:58.932056 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.932032 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6zp92_28f7079e-208d-4aea-ad43-eae93dffef7d/node-exporter/0.log" Jul 28 18:32:58.967574 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.967537 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6zp92_28f7079e-208d-4aea-ad43-eae93dffef7d/kube-rbac-proxy/0.log" Jul 28 18:32:58.991553 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:58.991519 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6zp92_28f7079e-208d-4aea-ad43-eae93dffef7d/init-textfile/0.log" Jul 28 18:32:59.180641 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.180612 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-jzn92_f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da/kube-rbac-proxy-main/0.log" Jul 28 18:32:59.204876 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.204843 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-jzn92_f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da/kube-rbac-proxy-self/0.log" Jul 28 18:32:59.228737 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.228711 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-jzn92_f7dc4332-8ef9-41f0-83d0-d2cf40ceb1da/openshift-state-metrics/0.log" Jul 28 18:32:59.261490 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.261465 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fb08721-54bb-4ad0-aa2e-ba36ebe731a0/prometheus/0.log" Jul 28 18:32:59.287664 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.287635 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fb08721-54bb-4ad0-aa2e-ba36ebe731a0/config-reloader/0.log" Jul 28 18:32:59.312517 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.312489 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fb08721-54bb-4ad0-aa2e-ba36ebe731a0/thanos-sidecar/0.log" Jul 28 18:32:59.337413 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.337389 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fb08721-54bb-4ad0-aa2e-ba36ebe731a0/kube-rbac-proxy-web/0.log" Jul 28 18:32:59.364399 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.364373 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fb08721-54bb-4ad0-aa2e-ba36ebe731a0/kube-rbac-proxy/0.log" Jul 28 18:32:59.390613 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.390585 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fb08721-54bb-4ad0-aa2e-ba36ebe731a0/kube-rbac-proxy-thanos/0.log" Jul 28 18:32:59.416834 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.416809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fb08721-54bb-4ad0-aa2e-ba36ebe731a0/init-config-reloader/0.log" Jul 28 18:32:59.449058 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.448973 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-24f2v_f0af88e5-3201-45a1-8ac3-fc51cdc8914f/prometheus-operator/0.log" Jul 28 18:32:59.471006 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.470976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-24f2v_f0af88e5-3201-45a1-8ac3-fc51cdc8914f/kube-rbac-proxy/0.log" Jul 28 18:32:59.497942 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.497907 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-sbzkv_8aedc707-345f-4645-9935-c1afe6cd5d01/prometheus-operator-admission-webhook/0.log" Jul 28 18:32:59.528294 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.528263 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bd555bcf8-jdj52_c996fa72-bf21-4f6a-b4e0-89543925ea6c/telemeter-client/0.log" Jul 28 18:32:59.553267 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.553241 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bd555bcf8-jdj52_c996fa72-bf21-4f6a-b4e0-89543925ea6c/reload/0.log" Jul 28 18:32:59.582939 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.578024 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bd555bcf8-jdj52_c996fa72-bf21-4f6a-b4e0-89543925ea6c/kube-rbac-proxy/0.log" Jul 28 18:32:59.612515 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.612486 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f7f7b7b8-fq4ws_c093872e-6524-4834-9b1f-1e2dbd258945/thanos-query/0.log" Jul 28 18:32:59.637054 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.637023 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f7f7b7b8-fq4ws_c093872e-6524-4834-9b1f-1e2dbd258945/kube-rbac-proxy-web/0.log" Jul 28 18:32:59.663785 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.663759 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f7f7b7b8-fq4ws_c093872e-6524-4834-9b1f-1e2dbd258945/kube-rbac-proxy/0.log" Jul 28 18:32:59.687327 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.687300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f7f7b7b8-fq4ws_c093872e-6524-4834-9b1f-1e2dbd258945/prom-label-proxy/0.log" Jul 28 18:32:59.711615 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.711535 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f7f7b7b8-fq4ws_c093872e-6524-4834-9b1f-1e2dbd258945/kube-rbac-proxy-rules/0.log" Jul 28 18:32:59.735204 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:32:59.735154 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f7f7b7b8-fq4ws_c093872e-6524-4834-9b1f-1e2dbd258945/kube-rbac-proxy-metrics/0.log" Jul 28 18:33:00.528998 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.528959 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h"] Jul 28 18:33:00.529556 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.529537 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6956230-d574-44ac-87ff-cd5768a1ccda" containerName="revoke-keys" Jul 28 18:33:00.529648 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.529559 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6956230-d574-44ac-87ff-cd5768a1ccda" containerName="revoke-keys" Jul 28 18:33:00.529702 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.529667 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6956230-d574-44ac-87ff-cd5768a1ccda" containerName="revoke-keys" Jul 28 18:33:00.533064 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.533043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.534941 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.534918 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sx94q\"/\"default-dockercfg-t679m\"" Jul 28 18:33:00.535532 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.535513 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sx94q\"/\"kube-root-ca.crt\"" Jul 28 18:33:00.535642 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.535562 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sx94q\"/\"openshift-service-ca.crt\"" Jul 28 18:33:00.542622 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.542598 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h"] Jul 28 18:33:00.600494 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.600454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-sys\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.600676 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.600551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-podres\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.600676 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.600578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkqr\" (UniqueName: \"kubernetes.io/projected/8618cf43-7c1d-4449-bff9-d51cece18849-kube-api-access-wjkqr\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.600676 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.600651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-proc\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.600801 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.600690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-lib-modules\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.701978 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.701939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-podres\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.701978 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.701976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkqr\" (UniqueName: \"kubernetes.io/projected/8618cf43-7c1d-4449-bff9-d51cece18849-kube-api-access-wjkqr\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.702274 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.702005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-proc\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.702274 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.702027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-lib-modules\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.702274 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.702074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-sys\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.702274 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.702092 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-proc\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.702274 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.702130 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-podres\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.702274 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.702160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-sys\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.702274 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.702240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8618cf43-7c1d-4449-bff9-d51cece18849-lib-modules\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.710830 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.710802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkqr\" (UniqueName: \"kubernetes.io/projected/8618cf43-7c1d-4449-bff9-d51cece18849-kube-api-access-wjkqr\") pod \"perf-node-gather-daemonset-mj42h\" (UID: \"8618cf43-7c1d-4449-bff9-d51cece18849\") " pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.842816 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.842704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:00.981027 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:00.980828 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h"] Jul 28 18:33:00.983415 ip-10-0-139-157 kubenswrapper[2576]: W0728 18:33:00.983386 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8618cf43_7c1d_4449_bff9_d51cece18849.slice/crio-3925fb7ace73626b3e59c5cd80dfe0d9239c68e4eb1f6c737bd33a57c88cc266 WatchSource:0}: Error finding container 3925fb7ace73626b3e59c5cd80dfe0d9239c68e4eb1f6c737bd33a57c88cc266: Status 404 returned error can't find the container with id 3925fb7ace73626b3e59c5cd80dfe0d9239c68e4eb1f6c737bd33a57c88cc266 Jul 28 18:33:01.803623 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:01.803582 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" event={"ID":"8618cf43-7c1d-4449-bff9-d51cece18849","Type":"ContainerStarted","Data":"8067c8b91a7038484d3ef945f86b1fe7d64fc8166a60695bc554fe2e598d5644"} Jul 28 18:33:01.804115 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:01.803638 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" event={"ID":"8618cf43-7c1d-4449-bff9-d51cece18849","Type":"ContainerStarted","Data":"3925fb7ace73626b3e59c5cd80dfe0d9239c68e4eb1f6c737bd33a57c88cc266"} Jul 28 18:33:01.804115 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:01.803662 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:01.822250 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:01.822199 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" podStartSLOduration=1.82218324 podStartE2EDuration="1.82218324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:33:01.820163745 +0000 UTC m=+3506.792671295" watchObservedRunningTime="2026-07-28 18:33:01.82218324 +0000 UTC m=+3506.794690782" Jul 28 18:33:02.114858 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:02.114760 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-49w5r_7bf7b156-f793-493b-9f1a-91afef4fb73f/download-server/0.log" Jul 28 18:33:03.920129 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:03.920101 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z6ph2_df9b2c95-b7de-43dd-83c0-1f49969d5d25/dns/0.log" Jul 28 18:33:03.944978 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:03.944947 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z6ph2_df9b2c95-b7de-43dd-83c0-1f49969d5d25/kube-rbac-proxy/0.log" Jul 28 18:33:04.031710 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:04.031685 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-prxq5_de3de89b-e0c9-4347-b778-e7893bef0f5e/dns-node-resolver/0.log" Jul 28 18:33:04.627410 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:04.627376 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-67cb9_7785bf5b-11ef-44ba-a530-f83e062952a9/node-ca/0.log" Jul 28 18:33:05.844388 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:05.844352 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b8f9cc6c-ssgcc_287d6281-c6a8-4cdf-a0c9-187778f10002/kube-auth-proxy/0.log" Jul 28 18:33:06.931793 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:06.931763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-shzch_48f71842-cc3b-4b31-be38-a7687228407a/serve-healthcheck-canary/0.log" Jul 28 18:33:07.553923 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:07.553890 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8qb72_2158ecfe-9984-4ecb-9a13-7c02d3cad353/kube-rbac-proxy/0.log" Jul 28 18:33:07.576945 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:07.576912 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8qb72_2158ecfe-9984-4ecb-9a13-7c02d3cad353/exporter/0.log" Jul 28 18:33:07.602389 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:07.602360 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8qb72_2158ecfe-9984-4ecb-9a13-7c02d3cad353/extractor/0.log" Jul 28 18:33:07.819674 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:07.819595 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sx94q/perf-node-gather-daemonset-mj42h" Jul 28 18:33:10.840773 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:10.840655 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7c5665b5-sc52l_1d066b5a-7b39-4c48-b5c1-58bb3befbd85/manager/0.log" Jul 28 18:33:10.862563 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:10.862533 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7c5665b5-sc52l_1d066b5a-7b39-4c48-b5c1-58bb3befbd85/copy-manifests/0.log" Jul 28 18:33:11.062872 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:11.062824 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b8f9cc6c-ssgcc_287d6281-c6a8-4cdf-a0c9-187778f10002/kube-auth-proxy/0.log" Jul 28 18:33:12.223782 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:12.223747 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8c847c87f-gscjr_f8909975-4b7a-44aa-959e-7d3376e735ea/manager/0.log" Jul 28 18:33:12.858927 ip-10-0-139-157 kubenswrapper[2576]: I0728 18:33:12.858895 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b8f9cc6c-ssgcc_287d6281-c6a8-4cdf-a0c9-187778f10002/kube-auth-proxy/0.log"